@angerman @mpickering

So it seems to work well and actually fixes the original issue, so I’m going to do the same to other compilers.

@GrahamcOfBorg eval

I'm generally super duper happy about these changes, except for the pkgs/top-level/haskell-packages.nix ones. The idea with there is that instead of doing buildCompiler everywhere, we can just do buildHaskellPackages.compiler, buildHaskellPackages.hscolour, etc. This is just like how we do buildPackages.stdenv.cc instead of a hypothetical stdenv.buildCC or something. bootPkgs i think also should be kept, and instead the ghcs should do bootPkgs.hscolour instead of taking a seprate hscolour argument.

@Ericson2314 Thanks for the comments! I think I am starting to understand what is going on, but I still have so many questions!

1. The idea with there is that instead of doing buildCompiler everywhere, we can just do buildHaskellPackages.compiler

   This sounds to me more or less like passing pkgs and buildPackages explicitly and then letting the expr get dependencies it wants manually instead of using callPackage 🤔. I can’t say that I find this idea bad, it’s quite the opposite, I think I like it, but it seems to contradict current nixpkgs practices. Don’t you think so?

2. The generic builder builds both the package and its Setup.hs. In order to build the package it needs libs from pkgs and build tools from pkgs.buildPackages. In order to build Setup.hs it needs libs from pkgs.buildPackages and build tools from pkgs.buildPackages.buildPackages. I don’t think it currently has access to the latter, but, if I understand correctly, it works this around by using those six attributes of mkDerivation. So, it takes nativeGhc from buildHaskellPackages, which means it is shifted one back and then puts it into pkgsBuildBuild which, iiuc, shifts it two more back, which gives three, doesn’t it? And we wanted just two. Why even bother getting nativeGhc from buildHaskellPackages if it is the same ghc that we were given anyway?

   buildHaskellPackages = bh.packages.ghc822;
   ghc = bh.compiler.ghc822;
   

   What also makes me worry is that we use various nativeGhc attributes (assuming, I guess, that, in theory it can be a separate compiler than ghc, even though it can’t as the code above shows), but then we use these attributes from a “-1-compiler” with a “-3-compiler” when what we actually want is a “-2-compiler”. I have a feeling that I am missing something very important.

3. I completely don’t understand the different use of buildHaskellPackages. In GHC package sets it is just a build version of the same set. In GHCJS package sets it is a build version of, mmm, the set of packages used to build a matching(?) version of GHC? I don’t get it at all. :(

Given that I don’t really understand what is going on, I’ll just revert those changes that are irrelevant, as, I think, we can do well with having a heavily overloaded ghc attribute in the package sets, at least for some time.

@kirelagin I'll post more when I'm not on my phone (happy to also chat on IRC today), but the short answer to all of those is https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/splice.nix hasn't been done for Haskell packages sets, so all the usual magic of getting the right version doesn't work. I do hate splicing, but maybe I just need to make a newScopeWithSlicing so Haskell and all the other users of newScope will not wet the bed with cross.

CC @matthewbauer

IRC sounds great, unfortunately, I’m afraid I won’t be available today, we can probably try tomorrow.

If splicing is not implemented for haskell packages, it means that a wrong compiler is used for Setup.hs as it is shifted only 1 back, not 2. nativeGhc should be coming from bootPackages.bootPackages.

@kirelagin I wish your second comment was correct! The problem is haskellPackageSet.ghc today is a lie, it is not the GHC built in that package set, as would be consistent with everything else, but the GHC used to build that package set.

stdenv has the same issue: pkgs.stdenv.cc is not the gcc or clang from pkgs, but the gcc or clang from buildPackages. But least stdenv.cc has a "meta-package"-sounding name (to use the Debian term). That's one of the reasons I really like your ghc-to-compiler rename: it also gives it a "meta-package"-sounding name for consistency with stdenv. Then also we can make buildPackages.ghc normal again if we every get ghc building like a normal package, too 🤞 🤞 🤞 🤞. Put another way, we need some sort of alias / can't just hard-code buildPackages.ghc were as we may be using another compiler, like GHCJS (buildPackages.ghcjs someday).

This is part of the reason why I changed it to passing the compiler explicitly – this way it is easier to track what is going on instead of trying to figure out which ghc comes from which package set and what it means.

Regarding the name, I would really prefer those “meta” names to have something special in them, e.g. _compiler (or just not be there and be passed explicitly), because it works only until someone uploads a package called compiler to Hackage 🤷‍♂️.

Another option would be not to pass the compiler explicitly and not put it into the package set, but instead pass a package-set-selector (for example just the name, e.g. "ghc822") and pkgs to let the expression do pkgs{,.buildPackages}.haskell.{compiler,packages}."${selector}" whenever it needs a package or the compiler. What do you think?

@kirelagin I tried not adding buildHaskellPackages to the set for similar reasons, but I've decided that was kind of too annoying and want to make it part of the package set after all, so I think I lean towards the _compiler option.

@kirelagin Are you / Serokell still interested in this sort of thing? I lay out in https://gitlab.haskell.org/ghc/ghc/issues/16503 how we're pretty close to being able to build a stage1 GHC that doesn't assume any bootstrapping plan, rather reading everything it needs from the settings file. That + some jerry-rigging in Nix along these lines would allow cross compiling to any platform. A build = host "cross compile" then in turn allows building another "stage 1" compiler (as far as GHC's own build system is concerned) that when told in settings to enable GHCi, TH, etc won't crash but work properly; i.e. is as good as a normal stage 2 proper native GHC build.

@Ericson2314 I, personally, am very interested and Serokell, in principle, too, however I’ll need to check whether we can actually devote any effort to this at the moment.
My current view on the subject is rather pessimistic: I feel that it is easier to rewrite all of nixpkgs Haskell infrastructure form scratch, migrating it to cabal-new and resolving other issues at the same time. However, I’ll take a look at the issue you mentioned and see if we can fix boot libraries in a less intrusive way...

Yes this is in part why I was bring up the GHC side of things. The more GHC can optionally "delegate" bootstrapping to the package manager (i.e. the Nix haskell infra), the better job the package manager can do.

If you are referring to https://github.com/input-output-hk/nix-tools, I agree we should be using that :). I guess I might be a bit more optimistic in that I think once GHC has its stuff order even the current infra could also take advantage of it too, but regardless, GHC is the choke-point here.

The documentation for haskell.nix (used to be referred to as nix-tools) is at https://input-output-hk.github.io/haskell.nix/

Notably we are using it mostly with a somewhat crude hack to make lib:ghc reinstallable (https://gitlab.haskell.org/ghc/ghc/merge_requests/490). I'm waiting for @Ericson2314 to clean up his settings file managing and then rebase it ontop of that. It should make the reinstallable lib:ghc even easier.

I've got the vast majority of the multi-target work done, and the rest open PRs. That's meant I've been able to start shuffling the build system to ensure no regressions by construction, and also make building things separately more robust: https://gitlab.haskell.org/ghc/ghc/merge_requests/1897/diffs

The end is in sight!