[WIP] mingw-w64 for haskell cross compilation with ghc. #37254

angerman · 2018-03-17T15:41:39Z

Motivation for this change

This includes all the necessary changes to nixpkgs, that allow me to cross compile haskell to
windows via mingw-w64. It is not yet sufficient to allow cross compilation of Template Haskell.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

angerman · 2018-03-17T15:42:55Z

/cc @Ericson2314

angerman · 2018-03-17T15:44:13Z

pkgs/development/compilers/ghc/8.2.1-binary.nix

@@ -4,7 +4,7 @@
 }:

 # Prebuilt only does native
-assert stdenv.targetPlatform == stdenv.hostPlatform;
+# assert stdenv.targetPlatform == stdenv.hostPlatform;


While disabling this check, allows most of the cross compilation to succeed on macOS, I believe this is also the reason why we end up with mixed (build and target) packages in the stdenv eventually.

angerman · 2018-03-17T15:45:57Z

pkgs/development/compilers/ghc/head.nix

@@ -161,7 +175,11 @@ stdenv.mkDerivation rec {
  # zsh and other shells are smart about `{ghc}` but bash isn't, and doesn't
  # treat that as a unary `{x,y,z,..}` repetition.
  postInstall = ''
-    paxmark m $out/lib/${name}/bin/${if targetPlatform != hostPlatform then "ghc" else "{ghc,haddock}"}


This change is motivated by the fact that we do not necessarily build haddock with quick flavours. As such we fall over trying to paxmark a file that does not exist.

Sounds good-- probably can collapse this to just {ghc,haddock} if we check for existence anyway.

angerman · 2018-03-17T15:46:37Z

pkgs/development/haskell-modules/generic-builder.nix

    "--with-strip=${stdenv.cc.bintools.targetPrefix}strip"
-  ] ++ (if isHaLVM then [] else ["--hsc2hs-options=--cross-compile"]);
+  ] ++ (if isHaLVM then [] else ["--hsc2hs-option=--cross-compile"
+                                 "--hsc2hs-option=--via-asm"]);


--via-asm is a rather recent feature in hsc2hs that is part of ghc > 8.4

angerman · 2018-03-17T15:47:28Z

pkgs/os-specific/darwin/cctools/port.nix

@@ -12,7 +12,7 @@ let
    "${targetPlatform.config}-";
 in

-assert targetPlatform.isDarwin;
+assert stdenv.isDarwin;


This could potentially also be the reason for the incorrect package set on darwin.

Ericson2314 · 2018-03-20T18:46:46Z

pkgs/development/compilers/ghc/head.nix


 , version ? "8.5.20180118"
+, ghcRevision ? "e1d4140be4d2a1508015093b69e1ef53516e1eb6"
+, ghcSha256 ? "1gdcr10dd968d40qgljdwx9vfkva3yrvjm9a4nis7whaaac3ag58"
+, ghcFlavour ? ""


Better to do ghcFlavour ? if targetPlatform != hostPlatform then "perf-cros" else ""

dtzWill · 2018-03-22T11:33:33Z

pkgs/development/compilers/ghc/head.nix

+  enableShared ? (targetPlatform == hostPlatform && !targetPlatform.isWindows)
+
+, # Whetherto build terminfo.
+  enableTerminfo ? !targetPlatform.isWindows


Does terminfo/ncurses not work on windows?

If it was failing to find ncurses headers/libs see the fix on my musl PR (remove attempt to specify where to find them).

In the case of cross compiling to windows, we don't want terminfo, the GHC build system disables terminfoin that case anyway.

It only disables it in some cross build "flavour"s, which we don't have to use as-is. Works for me building cross-aarch64-musl and such. Anyway sounds like you're saying windows in particular shouldn't use this, which I'll take your word for.

dtzWill · 2018-03-22T11:35:34Z

pkgs/development/haskell-modules/generic-builder.nix

+          setupConfigureFlags+=" --extra-lib-dirs=$p/lib"
+        fi
+      done
+      ${nativeGhcCommand}-pkg --${packageDbFlag}="$setupPackageConfDir" recache


dtzWill · 2018-03-22T14:48:18Z

If you're touching cctools, #37606 might have some useful bits.

peti · 2018-03-23T08:46:41Z

pkgs/development/compilers/ghc/head.nix

-  enableShared ? true
+  #
+  # We don't do this by default if target != host, or if we target windows.
+  enableShared ? (targetPlatform == hostPlatform && !targetPlatform.isWindows)


I believe that the description of this option is actually quite incorrect and misleading. Setting this to true has no effect on whether ghc builds shared libraries or not. Rather, it affects the code that's used by ghci to load compiled objects for interpretation. There's a good chance that this setting is completely ignored on Windows anyway. Could you please check whether the change you've made here is actually necessary?

enable* is about how the current package is linked. I think this does just that. That this also affects how GHCi works is just a side affect due to GHCi's fragile interpretation.

dtzWill · 2018-04-21T13:01:42Z

Beep boop, how's this going?

angerman · 2018-04-21T14:06:09Z

@dtzWill this does work as-is (I've used this to build dhall-json among others...). I did however get severely frustrated with the conditionals missing in cabal2nix generated cabal expressions. I did look into making cabal2nix generate better expressions, but learned that what I wanted was a bad fit for what cabal2nix does. And I don't want to disrupt any working solution in nix.

So I ended up writing Cabal2Nix, which really needs a better name. I now have hackage.nix, stackage.nix as well as some glue code in haskell.nix. This is pretty much work-in-progress. While it does allow me to build some trivial hs-hello package, it is nowhere near polished yet, and will likely see some additional changes.

I believe it will provide a much better basis for cross compilation haskell packages.

Ericson2314 · 2018-04-22T00:43:31Z

@angerman that sounds fascinating but I think your links are broken?

angerman · 2018-04-22T06:54:00Z

@Ericson2314 I see... Here are the full links:

... thanks GitHub.

kirelagin · 2018-04-29T21:51:04Z

pkgs/development/haskell-modules/setup.nix

+    # Add the setupHaskellDepends to a custom package-db,
+    # which ghc will be provided with.
+
+    for p in ${lib.escapeShellArgs setupHaskellDepends}; do


Here propagated setup dependencies are not taken into account. That is, your pkgdb will contain only immediate setup dependencies but their transitive dependencies will not be in it, so it won’t compile.

kirelagin · 2018-04-29T21:54:23Z

pkgs/development/haskell-modules/generic-builder.nix

+  configureSetup = buildHaskellPackages.setup { setupHs = ''
+  import Distribution.Simple
+  main = defaultMainWithHooks autoconfUserHooks
+'';


inherit setupHaskellDepends, I guess.

kirelagin · 2018-04-29T22:03:28Z

pkgs/development/haskell-modules/setup.nix

+
+  phases = [ "buildPhase" ];
+
+  buildInputs = [ ghc ] ++ setupHaskellDepends;


I think ghc should go to nativeBuildInputs.

For cross compilation, we don't need it, it's deferred to the Setup derivation. When ghc is th enativeGhc, we don't need to add it to the buildInputs either.

nixpkgs#37012 and nixpkgs#37707 introduces the setup-hooks for libiconv, which inject `-liconv` into the `NIX_LDFLAGS`. This breaks horribly on windows where the linker end up having no idea how to linke `-liconv`. The configure.ac file specifically ignores libiconv on windows.

This reverts commit 7f86470.

Nix cc logic relies on passing /lib for each dependency as a library search path. This makes little sense for haskell packages, as their libs are nested deeply in the /lib tree. Also GHC will generate the right library serach paths on its own from the package database entries. The larger issue though is that with one search path entry for each library, and a similar one generated by GHC for each dependency will eventually overflow the command line argument limits. As such we do *not* put haskell libraries into $out anymore.

angerman requested a review from peti as a code owner March 17, 2018 15:41

angerman commented Mar 17, 2018

View reviewed changes

GrahamcOfBorg added 6.topic: haskell 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Mar 17, 2018

bjornfor added the 6.topic: cross-compilation Building packages on a different sort platform than than they will be run on label Mar 19, 2018

Ericson2314 reviewed Mar 20, 2018

View reviewed changes

angerman force-pushed the feature/ghc-cross-mingw branch from c8f682e to de9b33e Compare March 21, 2018 23:36

angerman requested a review from FRidh as a code owner March 21, 2018 23:36

GrahamcOfBorg added 6.topic: python 8.has: package (new) labels Mar 21, 2018

angerman mentioned this pull request Mar 21, 2018

ghc-{8.2.2,8.4.2}: fixups mostly for 2-stage cross, musl #37598

Merged

8 tasks

dtzWill reviewed Mar 22, 2018

View reviewed changes

matthewbauer added the 2.status: work-in-progress label Mar 22, 2018

peti requested changes Mar 23, 2018

View reviewed changes

bgamari mentioned this pull request Apr 24, 2018

Cross-compilation of Haskell packages with custom setup program #39417

Open

kirelagin mentioned this pull request Apr 28, 2018

haskell: Setup.hs should not be compiled with the same pkgdb as the package itself #39646

Closed

kirelagin suggested changes Apr 29, 2018

View reviewed changes

kirelagin reviewed Apr 29, 2018

View reviewed changes

angerman mentioned this pull request May 1, 2018

haskell: build Setup.hs with a separate pkgdb #39735

Closed

angerman and others added 26 commits May 16, 2018 16:42

Adds custom Setup derivation.

6c7fb30

use/provide setup in package sets.

2a8ded3

use Setup derivation when cross compiling.

56d28ed

do not add nativeGhc to the buildInputs

dcb0cb8

For cross compilation, we don't need it, it's deferred to the Setup derivation. When ghc is th enativeGhc, we don't need to add it to the buildInputs either.

no -staticlib for windows.

fcb8be5

Adds more asserts

cd79349

more logging.

2276ff8

Revert incorrect naming.

55f320f

Use buildpackages for nix and nix-prefetch-scripts

5666ec8

used stdenv.targetPlatform.isDarwin and not stdenv.isDarwin.

a302505

fix python27

093afd8

WIP

ca6cf4a

fix lz4 cross-build

a7f9517

make rocksdb use cmake (doesnt work)

3443dd3

fix libuv

45e4258

rocksdb (don't fully work yet...)

f4f044f

proper mingw names

3c032a1

revert rocksdb

7f86470

cleanup mingw

2cce00a

Adds preliminary gcc-ng infra.

728ada5

Adds wine (darwin)

cd3b3de

make bulding libgcc work.

1353a6e

Revert "revert rocksdb"

b1adfd1

This reverts commit 7f86470.

use gcc7-ng-msvcrt by default.

3c044d2

bump

68d2d15

angerman force-pushed the feature/ghc-cross-mingw branch from 98cff92 to 68d2d15 Compare June 8, 2018 15:46

angerman requested a review from nbp as a code owner June 8, 2018 15:46

angerman closed this Jul 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] mingw-w64 for haskell cross compilation with ghc. #37254

[WIP] mingw-w64 for haskell cross compilation with ghc. #37254

angerman commented Mar 17, 2018

angerman commented Mar 17, 2018

angerman Mar 17, 2018

angerman Mar 17, 2018

dtzWill Mar 22, 2018

angerman Mar 17, 2018

angerman Mar 17, 2018

Ericson2314 Mar 20, 2018

dtzWill Mar 22, 2018

angerman Mar 22, 2018

dtzWill Mar 22, 2018

dtzWill Mar 22, 2018 •

edited

dtzWill commented Mar 22, 2018

peti Mar 23, 2018

Ericson2314 May 1, 2018

dtzWill commented Apr 21, 2018

angerman commented Apr 21, 2018 •

edited

Ericson2314 commented Apr 22, 2018

angerman commented Apr 22, 2018

kirelagin Apr 29, 2018

kirelagin Apr 29, 2018

kirelagin Apr 29, 2018


		phases = [ "buildPhase" ];

		buildInputs = [ ghc ] ++ setupHaskellDepends;

[WIP] mingw-w64 for haskell cross compilation with ghc. #37254

[WIP] mingw-w64 for haskell cross compilation with ghc. #37254

Conversation

angerman commented Mar 17, 2018

Motivation for this change

Things done

angerman commented Mar 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtzWill Mar 22, 2018 • edited

Choose a reason for hiding this comment

dtzWill commented Mar 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtzWill commented Apr 21, 2018

angerman commented Apr 21, 2018 • edited

Ericson2314 commented Apr 22, 2018

angerman commented Apr 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtzWill Mar 22, 2018 •

edited

angerman commented Apr 21, 2018 •

edited