Paging @samueldr

@dhess I wonder if there's a way to control the number passed to -j in the build process, without affecting the number passed to make. It'd be great if we could make all the ghc --make calls avoid -j categorically.

But overall I'm happy about this change :)

The GHC build system uses -j with --make all over the place. Reverting this change breaks the GHC build on my aarch64 boards. As discussed on the ticket, this doesn't afflict all microarchitectures, but if you are affected, whether or not the build succeeds is a dice roll.

I would much rather have a build that works every time, rather than a build that fails on my machines but works on Hydra. Maybe I'm the only one unlucky enough to be using the afflicted hardware?

@TravisWhitaker I thought -j and --make are only used in building ghc-cabal and running the test suite. Where else is it used? We should probably disable the test quite though, unless we can figure out how to limit the -j for just that.

@TravisWhitaker The entire community benefits from having successful Hydra builds of GHC on aarch64, so I think that's the right tradeoff, even if it causes an inconvenience for you, personally. Though it's a pain, you can override ghc843 for your own builds with an overlay.

However, the point is moot (for now) as it appears that even Hydra is unable to build ghc843 reliably on aarch64 since the initial successful build:

https://hydra.nixos.org/build/83144215

It's timing out after 10 hours, which is interesting, because before this change, when ghc843 was being built with enableParallelBuilding disabled, it always took > 10 hours and therefore Hydra killed it mid-build. (Last I checked, the NixOS Hydra is configured to kill builds after 10 hours.)

The first successful build (https://hydra.nixos.org/build/82810083), just after this commit was made, took 8h23m, so it made the cut. But it might be the case that we just got lucky with that build, and that subsequent builds with enableParallelBuilding enabled have triggered the bug, causing the build to hang and Hydra to terminate it after 10 hours.

It's also possible that the builds were going fine but just took longer than the first successful build and didn't make the 10 hour cutoff.

So, either way, it appears that this change was not successful in its goal of producing consistent GHC Hydra builds on aarch64.

I will say, though, that on my own personal Hydra, I'm doing a daily build of 3 of my own Haskell packages using ghc843 from a daily Nixpkgs master snapshot, using the NixOS community aarch64 builder. These daily builds are working fine, so I'm not sure why they're succeeding when the official NixOS ones are not. I'll look into it.

The entire community benefits from having successful Hydra builds of GHC on aarch64, so I think that's the right tradeoff, even if it causes an inconvenience for you, personally.

With enableParallelBuilding = true, this derivation is nondeterministic in a critically bad way. If a derivation needed to be made nondeterministic to enable Hydra builds in a more benign way (i.e. something less than memory corruption), I'd certainly agree. However, unless these GHC binaries are passing validation (IIRC full validation isn't run, but perhaps it is) or we know for sure that the microarchitecture Hydra is using to build these is unaffected, I wouldn't trust the binaries in these packages.

I should've qualified "my" more carefully; my organization has its own Nix infrastructure and GHC tweaks, so we aren't using these binaries anyway. In general, I think a person with afflicted hardware would rather wait for a working binary to build, rather than download one from Hydra that might be arbitrarily broken due to this GHC bug.

It's timing out after 10 hours, which is interesting, because before this change, when ghc843 was being built with enableParallelBuilding disabled, it always took > 10 hours and therefore Hydra killed it mid-build.

As I wrote in the GHC bug report, a common failure mode is a threaded Haskell program hanging forever, waiting on a futex. Perhaps this is the culprit?

I will say, though, that on my own personal Hydra, I'm doing a daily build of 3 of my own Haskell packages using ghc843 from a daily Nixpkgs master snapshot, using the NixOS community aarch64 builder.

You might be running on unafflicted hardware (which seems to be any microarch without dynamic execution) or you've flipped three heads in a row.

I thought -j and --make are only used in building ghc-cabal and running the test suite. Where else is it used? We should probably disable the test quite though, unless we can figure out how to limit the -j for just that.

IIRC it's also used when building the boot packages, but I could be wrong.

@dhess Wait... The log of that failed build ends the same way as a successful build... Is it possible something irrelevant is hanging for hours? It is not exhibiting the panic caused by -j. There is no panic message in that log. @TravisWhitaker so I don't think this is due to enableParallelBuilding (again, -j should only be used in building ghc-cabal, and maybe some test suites).

EDIT:

As I wrote in the GHC bug report, a common failure mode is a threaded Haskell program hanging forever, waiting on a futex. Perhaps this is the culprit?

It can't be because then we wouldn't be getting to the installPhase, let alone the fixupPhase.

@dhess I wonder if we could get Hydra to build it with #45744 so we can see how long each phase is actually taking.

@ElvishJerricco Does it always panic? I was under the impression that it sometimes just hangs indefinitely. tmobile on the GHC Trac, who originally filed the issue (https://ghc.haskell.org/trac/ghc/ticket/15449), lists one of the manifestations as, "Compiler process sleeps indefinitely."

(edit Oh, perhaps @TravisWhitaker is tmobile ;)

@dhess Ah right. But it still wouldn't get to the installPhase or fixupPhase in that case. It'd hang in buildPhase or checkPhase or something.

It can't be because then we wouldn't be getting to the installPhase, let alone the fixupPhase.

Ah, agreed. It seems something else is going on...

@dhess Do the builders have auto-optimise-store turned on? I figure for an output as large as GHC, the post-build optimization of the path could potentially take a while on a slow core.

Does 8.4.4 support cross compiling from aarch64 to another architecture? I am getting some weird errors from cross-trunk:

https://hydra.nixos.org/build/83688270/nixlog/1/tail

ghc-stage1: panic! (the 'impossible' happened)
  (GHC version 8.4.4 for aarch64-unknown-linux):
        padLiveArgs -- i > regNum ??
CallStack (from HasCallStack):
  error, called at compiler/llvmGen/LlvmCodeGen/Base.hs:194:27 in ghc:LlvmCodeGen.Base

Actually this doesn't seem to be specific to aarch64, so probably not having to do with this PR.

Indeed, GHC 8.4.4 is totally broken on ARM.

https://ghc.haskell.org/trac/ghc/ticket/15780

I should have more time to look into this in a couple of weeks, but I'm completely slammed at the moment. Meanwhile, as recently as master from 4 days ago, ghc843 continues to work great on the NixOS aarch64 community builder, somehow.