Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ghc-{8.2.2,8.4.2}: fixups mostly for 2-stage cross, musl #37598

Merged
merged 1 commit into from Jun 18, 2018

Conversation

dtzWill
Copy link
Member

@dtzWill dtzWill commented Mar 21, 2018

  • grab patches from gentoo to fix various 2-stage cross build system
    bugs
  • explicitly set CrossCompilePrefix to the expected targetPrefix
    -- ensures everything has expected name and location
    -- fixes lack of prefix'ing when doing glibc -> musl "cross"
  • Stage1Only: only set if doing "true" cross
  • don't try to specify include/lib dirs for ncurses on cross
    -- only used by terminfo which actually removed the include option,
    and the lib option doesn't seem to do anything other than
    confuse the situation re:cross.

Fixes #37522

  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

Tested native x86_64, aarch64, cross-to-musl on those platforms,
and 'hello' :).

@dtzWill dtzWill requested a review from peti as a code owner March 21, 2018 21:08
@dtzWill
Copy link
Member Author

dtzWill commented Mar 21, 2018

cc @Ericson2314 @bgamari

@dtzWill
Copy link
Member Author

dtzWill commented Mar 21, 2018

Since I put this together for someone on IRC, here's a command for turning hours of CPU time into a musl-based haskell "Hello World":

$ NIX_PATH=nixpkgs=https://github.com/dtzWill/nixpkgs/archive/feature/ghc-cross-musl.tar.gz nix-build '<nixpkgs>' --arg crossSystem '{config="x86_64-unknown-linux-musl";}' -A haskell.packages.ghc841.hello 

😁

@Ericson2314
Copy link
Member

CC @angerman

@angerman
Copy link
Contributor

The BuildFlavour fix is not working because it misses the crucial import. That's part of the fixes in #37254. However, I still believe we might actually rather want to do

sed -E "s/^#BuildFlavour[ ]+= ${BuildFlavour}\$/BuildFlavour = ${BuildFlavour}/" < mk/build.mk.sample > mk/build.mk

That in turn should also set the Stage1Only values right.

@Ericson2314
Copy link
Member

If you two (and @bgamari too, maybe) could collaborate, that would be great!

@Ericson2314
Copy link
Member

Ericson2314 commented Mar 22, 2018

Historically, I've been able to cross compile fine without the gentoo patches, but I'll look through them to see what they offer.

@Ericson2314
Copy link
Member

Per haskell/cabal#4941 (comment) @23Skidoo would be interested in us producing official cabal-install binaries statically linked with musl. LMK what I can do to help with these PRs.

@nh2
Copy link
Contributor

nh2 commented Jun 12, 2018

Since I put this together for someone on IRC, here's a command for turning hours of CPU time into a musl-based haskell "Hello World":

@dtzWill Fantastic, this worked!

I have uploaded a built nix closure of the GHC built in that process here, so that people who don't want to wait hours to compile GHC can use my binary build:

https://github.com/nh2/nixpkgs/releases/tag/ghc-cross-musl-works

@nh2
Copy link
Contributor

nh2 commented Jun 13, 2018

The stuff in here allowed me to build some fully static Haskell executables:

https://github.com/nh2/static-haskell-nix

@nh2
Copy link
Contributor

nh2 commented Jun 13, 2018

@dtzWill
Copy link
Member Author

dtzWill commented Jun 13, 2018

@dtzWill Which of these branches is more up-to-date / should I use?

haha I honestly don't know. :/ sorry.

There were other deeper problems that made cross-compiling (musl or no) with GHC problematic,
I think the best solution for static builds would be to build musl-native (localSystem instead of crossSystem),
which requires having a "bootstrap" GHC that's also musl-native.
I was hoping to get this from these cross efforts, but I'm not sure how to create such a "bootstrap" package appropriately (or at all).

Or maybe cross-compilation with GHC works better now than it did, not sure.
Still building "native" seems less likely to be problematic if we can clear the bootstrap hurdle :).

@Ericson2314 Ericson2314 changed the title ghc-{8.2.2,8.4.1}: fixups mostly for cross, musl ghc-{8.2.2,8.4.1}: fixups mostly for 2-stage cross, musl Jun 13, 2018
@Ericson2314
Copy link
Member

Ah I finally understand this now. This stuff is for 2-stage cross, which wasn't attempted before. It's a somewhat odd use-case, but seems at least harmless to everything else. I'm happy merging this if you all want; just got rid of conflicts.

@Ericson2314
Copy link
Member

Actually @angerman I think recently submitted a better version of the last patch. I'm going to gate them all on the two-stage case.

* grab patches from gentoo to fix various 2-stage cross build system
  bugs
* explicitly set CrossCompilePrefix to the expected targetPrefix
  -- ensures everything has expected name and location
  -- fixes lack of prefix'ing when doing glibc -> musl "cross"
* Stage1Only: only set if doing "true" cross
* don't try to specify include/lib dirs for ncurses on cross
  -- only used by terminfo which actually removed the include option,
  and the lib option doesn't seem to do anything other than
  confuse the situation re:cross.

Fixes NixOS#37522
@nh2
Copy link
Contributor

nh2 commented Jun 13, 2018

@Ericson2314 The rebase seems to make GHC 8.2.2 unhappy: #37522 (comment)

@dtzWill
Copy link
Member Author

dtzWill commented Jun 13, 2018

@Ericson2314 thanks! What do you mean "2-stage cross", and why is it odd? (not challenging, just .. I don't remember this at all and am curious!)

@Ericson2314
Copy link
Member

Ericson2314 commented Jun 13, 2018

Normally GHC just builds a single cross compiler and that's this. Here GHC builds a cross compiler which builds a native compiler, a process that is only possible because the target platform is very close to the build platform (i.e. can run musl binaries on linux with glibc for most things). General compilation is limited to a single-stage build.

@angerman
Copy link
Contributor

@Ericson2314, which last patch are you referring to?

@angerman
Copy link
Contributor

I don’t think i’ve Done anything to hp2ps or unlit. Only to hsc2hs. I’m afraid you must be misremembering, @Ericson2314.

@nh2
Copy link
Contributor

nh2 commented Jun 14, 2018

With the version before the rebase, I managed to build a Haskell package with lots of dependencies (stack) fully statically: https://github.com/nh2/static-haskell-nix/tree/master/static-stack

Now we just have to get it working on the latest nixpkgs, and figure out the haddock problems (see here).

@Ericson2314
Copy link
Member

Ericson2314 commented Jun 14, 2018

@angerman Ah I was thinking of haskell/hsc2hs@738f366, which was inspired by those patches.

@Ericson2314 Ericson2314 merged commit 07ff2c2 into NixOS:staging Jun 18, 2018
@dtzWill
Copy link
Member Author

dtzWill commented Jun 18, 2018

(!!!)

~~it's happening!!~~

@Ericson2314 Ericson2314 changed the title ghc-{8.2.2,8.4.1}: fixups mostly for 2-stage cross, musl ghc-{8.2.2,8.4.2}: fixups mostly for 2-stage cross, musl Jun 18, 2018
@Ericson2314
Copy link
Member

Oh we need 8.4.3 now too.

@dtzWill
Copy link
Member Author

dtzWill commented Jun 18, 2018

FWIW build path for me seems to go through sqlalchemy, which is currently failing to build (failed test) w/musl but is fixed by bumping its version, see #42180 . At least ran into this trying to build 8.2.2.

@dtzWill
Copy link
Member Author

dtzWill commented Jun 25, 2018

@Ericson2314 any idea what the problem with 8.4.3 is? It builds (and seems to work when I manually invoke it on a trivial program), see #42455 (comment) for a bit more details.

Looks like the output doesn't have the right prefixes-- trying a quick fix that I don't expect to work but might do the trick... otherwise thoughts/suggestions are welcome :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants