A selection of cross-compiling fixes #39420

bgamari · 2018-04-24T18:53:57Z

Motivation for this change

Allow a variety of core packages to be cross-compiled. This is a batch of patches pulled from my larger batch in #30882. Many of the rest were kindly merged by @dtzWill. The few that remain after this will require a bit more work.

I've been using these to cross-compile ARMv7 NixOS images for several months now.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

bgamari · 2018-04-24T18:57:38Z

Sadly I can't build this with nox-review; it fails with Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS, presumably due to the very large size of the build.

dtzWill · 2018-04-24T19:09:06Z

Yay! And bumping initial heap size might help there?

Apologies for not checking myself, but just making sure: have you checked if these are still needed?

I think AR fixes, and curl fix (why do we set cc/cxx at all?) are needed, but itd be good to know what is/isn't. I trust they were needed when you wrote them but lots has changed :D.

On mobile, will look through these when at a computer next :).

Ericson2314 · 2018-04-24T19:16:34Z

pkgs/os-specific/linux/nfs-utils/default.nix

@@ -37,7 +39,10 @@ in stdenv.mkDerivation rec {
      "--with-systemd=$(out)/etc/systemd/system"
      "--enable-libmount-mount"
    ]
-    ++ lib.optional (stdenv ? glibc) "--with-rpcgen=${stdenv.glibc.bin}/bin/rpcgen";
+    ++ lib.optional (stdenv ? glibc) "--with-rpcgen=${stdenv.glibc.bin}/bin/rpcgen"
+    ++ stdenv.lib.optionals (stdenv.hostPlatform != stdenv.buildPlatform) [


This should be unconditional. But is it even still needed now that we export that environment variable?

Huh, indeed this doesn't appear to be necessary any more yet I thought I had already kicked out the patches that proved to be redundant.

I'll drop this patch.

Ericson2314 · 2018-04-24T19:19:20Z

pkgs/os-specific/linux/tcp-wrappers/default.nix

@@ -20,6 +20,7 @@ in stdenv.mkDerivation rec {
  prePatch = ''
    tar -xaf $debian
    patches="$(cat debian/patches/series | sed 's,^,debian/patches/,') $patches"
+    substituteInPlace Makefile --replace "AR	= ar" "AR = ${stdenv.cc.targetPrefix}ar"


I sort of like "AR ?= ar" if that works, but silly nit.

FWIW it might be best (if you disagree nevermind it's basically the same) if instead of patching the Makefile we just add "AR:=${stdenv.cc.targetPrefix}ar", or "AR:=$(AR)" actually, to makeFlags?

I have absolutely no opinion. I will happily do whatever you all deem best.

Wow, I am slightly embarrassed to admit that I had no idea that make variables given on the command-line override definitions in the Makefile. Yes, overriding seems like a significantly better option.

Or is this := overriding =?

To be clear, I didn't know this either, and I approve of @dtzWill's approach.

"AR:=$(AR)" is especially nifty.

Ericson2314 · 2018-04-24T19:20:13Z

pkgs/tools/security/rng-tools/default.nix

+  # For cross-compilation
+  configureFlags = [ "AR=${stdenv.cc.bintools.targetPrefix}ar" ];
+  postPatch = ''
+    substituteInPlace Makefile.in --replace "AR = ar" ""


Maybe "AR ?= ar" to be consistent with the above?

Also, since AR in the environment is already target-prefixed, are the configureFlags needed?

"AR:=$(AR)" per the above.

matthewbauer · 2018-04-24T19:25:47Z

pkgs/os-specific/linux/libnl/default.nix

@@ -19,6 +19,8 @@ stdenv.mkDerivation {
      sha256 = "0dd7xxikib201i99k2if066hh7gwf2i4ffckrjplq6lr206jn00r";
    });

+  enableParallelBuilding = true;


Not that this hurts anything - but I'm kind of amazed this would fix cross compiling. Usually parallel building breaks stuff. This is the first time I've heard of it fixing anything.

It helps all the time when cross-compiling*!

not fixing errors but making the rebuild/rebuild/rebuild go a bit faster and reduces the toll on your sanity!

not fixing errors but making the rebuild/rebuild/rebuild go a bit faster and reduces the toll on your sanity!

Couldn't have said it better myself.

bgamari · 2018-04-24T20:14:08Z

pkgs/os-specific/linux/nfs-utils/default.nix

@@ -37,7 +39,10 @@ in stdenv.mkDerivation rec {
      "--with-systemd=$(out)/etc/systemd/system"
      "--enable-libmount-mount"
    ]
-    ++ lib.optional (stdenv ? glibc) "--with-rpcgen=${stdenv.glibc.bin}/bin/rpcgen";
+    ++ lib.optional (stdenv ? glibc) "--with-rpcgen=${stdenv.glibc.bin}/bin/rpcgen"
+    ++ stdenv.lib.optionals (stdenv.hostPlatform != stdenv.buildPlatform) [


I'll drop this patch.

bgamari · 2018-04-24T20:18:26Z

pkgs/os-specific/linux/tcp-wrappers/default.nix

@@ -20,6 +20,7 @@ in stdenv.mkDerivation rec {
  prePatch = ''
    tar -xaf $debian
    patches="$(cat debian/patches/series | sed 's,^,debian/patches/,') $patches"
+    substituteInPlace Makefile --replace "AR	= ar" "AR = ${stdenv.cc.targetPrefix}ar"


Wow, I am slightly embarrassed to admit that I had no idea that make variables given on the command-line override definitions in the Makefile. Yes, overriding seems like a significantly better option.

bgamari · 2018-04-24T20:34:12Z

pkgs/tools/system/runit/default.nix

-    echo cc > conf-ld
+    echo ${stdenv.cc.targetPrefix}cc > conf-cc
+    echo ${stdenv.cc.targetPrefix}cc > conf-ld
+    export AR=${stdenv.cc.targetPrefix}ar


Hmm, I suppose this also isn't needed anymore.

Oh good point

dtzWill · 2018-04-24T21:07:34Z

All good, I surely know what it's like to sit on a huge stack of patches and test them over and over while splitting them out. And always there's a detail or two I overlooked! That I swear wasn't like that or was needed a month ago or in a cross-env ... hehe.

Anyway, thanks for pushing these through and look forward to next iteration! :)

bgamari · 2018-04-30T01:05:18Z

I've added two more low-risk fixes to this branch: glib and avahi.

Ericson2314 · 2018-05-01T03:05:58Z

@bgamari looks good except you actually added c0e0bc8 and 2400170?

bgamari · 2018-05-03T19:50:51Z

Fixed.

Ericson2314 · 2018-05-03T20:01:07Z

Thanks @bgamari

This package, like so many others, assumes that ar is unprefixed.

This is a long enough derivation to make this worthwhile.

By fixing toolchain paths

glib needs gettext for the build machine. Since gettext is also a propagatedBuildInput I'm assuming that it's also used at runtime.

dtzWill · 2018-05-03T21:27:52Z

lib/systems/for-meta.nix

+    { kernel = parse.kernels.linux; abi = parse.abis.gnu; }
+    { kernel = parse.kernels.linux; abi = parse.abis.gnueabi; }
+    { kernel = parse.kernels.linux; abi = parse.abis.gnueabihf; }
+  ];


Just curious: why is this not just matching on ABI's? It seems unintuitive for "this package runs on GNU" to mean "gnu/linux" (and as a recent PR addressed seems to have been abused to mean "linux" in many packages).

But would it be "bad" to not require the kernel be linux?

A good question. I was looking to minimize the potential risk incurred by my patch, but I'll admit that this definition is rather fishy.

Haha, understood re:conservative changes. I do the same thing all the time. cc @Ericson2314 -- do you know the answer?

Not a big deal ... :)

I'm fine either way. I made some changes upstream to GNU Config that are leading me to reconsider some of this stuff a bit. E.g. maybe ABI should be a per-kernel namespace.

http://lists.gnu.org/archive/html/config-patches/2018-04/threads.html

http://lists.gnu.org/archive/html/config-patches/2018-05/threads.html

bgamari · 2018-05-03T21:29:30Z

Yay!

dtzWill

These don't all seem to be needed--see comments.

If they're needed in your environment, I'd be curious to sort out why that is and for improving my own testing :).

dtzWill · 2018-05-03T21:37:25Z

pkgs/os-specific/linux/mmc-utils/default.nix

@@ -10,6 +10,8 @@ stdenv.mkDerivation rec {
    sha256 = "0hkdzc71pdnscbpdpgwljcchiyancarldjyd0w609sy18bky833x";
  };

+  makeFlags = "CC=${stdenv.cc.targetPrefix}cc";


I don't believe this is needed, here's cross-building to aarch64-musl:

$ nix build -f channel:nixos-unstable --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform-musl' mmc-utils $ ls -l result lrwxrwxrwx 1 will users 91 May 3 16:34 result -> /nix/store/g5izr5yd8sydw6dizmnj8y2mlvwx28a5-mmc-utils-2015-11-18-aarch64-unknown-linux-musl

(where unstable is based on master, and doesn't have this commit yet)

The same here.

dtzWill · 2018-05-03T21:38:07Z

pkgs/os-specific/linux/net-tools/default.nix

@@ -15,6 +15,8 @@ stdenv.mkDerivation rec {
    '';

  makeFlags = [
+    "CC=${stdenv.cc.targetPrefix}cc"
+    "AR=${stdenv.cc.targetPrefix}ar"


These don't appear to be needed either. Same command as above, but try building nettools.

Hmm, I could have sworn I cherry-picked this one for a reason but indeed I haven't seen a failure yet after reverting it. Very odd.

dtzWill · 2018-05-03T21:40:05Z

pkgs/tools/networking/curl/default.nix

-  CXX = "c++";
-  CXXCPP = "c++ -E";
+  CXX = "${stdenv.cc.targetPrefix}c++";
+  CXXCPP = "${stdenv.cc.targetPrefix}c++ -E";


This works as-is for me as well?

$ nix build -f channel:nixos-unstable --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform-musl' curl

I'll admit I didn't test this one. I had assumed that c++ would either not exist or refer to the host compiler.

That assumption does make sense to me. That this would work is very odd. Does it actually run or just build? I'd think just the latter.

dtzWill · 2018-05-03T21:41:22Z

pkgs/tools/security/rng-tools/default.nix

@@ -9,6 +9,9 @@ stdenv.mkDerivation rec {
    sha256 = "13h7lc8wl9khhvkr0i3bl5j9bapf8anhqis1lcnwxg1vc2v058b0";
  };

+  # For cross-compilation
+  makeFlags = [ "AR:=$(AR)" ];
+


Yes, this is needed! 😁

$ nix build -f channel:nixos-unstable --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform-musl' rng_tools 16:40:32 on 18-05-03 error: build of '/nix/store/4sh7pnz0vwacxm7a784fyf6q6cbzjhaw-rng-tools-5-aarch64-unknown-linux-musl.drv' on 'ssh://wdietz2@presto' failed: builder for '/nix/store/4sh7pnz0vwacxm7a784fyf6q6cbzjhaw-rng-tools-5-aarch64-unknown-linux-musl.drv' failed with exit code 2 builder for '/nix/store/4sh7pnz0vwacxm7a784fyf6q6cbzjhaw-rng-tools-5-aarch64-unknown-linux-musl.drv' failed with exit code 1; last 10 log lines: aarch64-unknown-linux-musl-gcc -DHAVE_CONFIG_H -I. -g -O2 -c fips.c rm -f librngd.a ar cru librngd.a fips.o /nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/bash: ar: command not found make[2]: *** [Makefile:440: librngd.a] Error 127 make[2]: Leaving directory '/build/rng-tools-5' make[1]: *** [Makefile:672: all-recursive] Error 1 make[1]: Leaving directory '/build/rng-tools-5' make: *** [Makefile:377: all] Error 2 builder for '/nix/store/4sh7pnz0vwacxm7a784fyf6q6cbzjhaw-rng-tools-5-aarch64-unknown-linux-musl.drv' failed with exit code 2 [0 built (1 failed), 0.0 MiB DL] error: build of '/nix/store/4sh7pnz0vwacxm7a784fyf6q6cbzjhaw-rng-tools-5-aarch64-unknown-linux-musl.drv' failed

Ericson2314 · 2018-05-03T22:16:12Z

@dtzWill Thanks for investigating, and my bad on the eager merge. I guess just revert those bits before staging evals again?

bgamari · 2018-05-04T03:55:03Z

Sorry for the confusion everyone! I could have sworn those two were needed.

bgamari force-pushed the cross-fixes-1 branch from 05e65f9 to dfe9f29 Compare April 24, 2018 18:55

Ericson2314 reviewed Apr 24, 2018

View reviewed changes

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 501+ labels Apr 24, 2018

matthewbauer reviewed Apr 24, 2018

View reviewed changes

bgamari commented Apr 24, 2018

View reviewed changes

bgamari force-pushed the cross-fixes-1 branch from dfe9f29 to c2731e3 Compare April 29, 2018 20:52

bgamari requested a review from nbp as a code owner April 29, 2018 20:52

GrahamcOfBorg added 10.rebuild-darwin: 501+ and removed 10.rebuild-darwin: 1-10 labels Apr 29, 2018

GrahamcOfBorg added 8.has: documentation 10.rebuild-darwin-stdenv labels Apr 30, 2018

bgamari force-pushed the cross-fixes-1 branch from 9d73046 to ea49097 Compare May 3, 2018 19:50

GrahamcOfBorg removed the 8.has: documentation label May 3, 2018

Ericson2314 approved these changes May 3, 2018

View reviewed changes

GrahamcOfBorg removed the 10.rebuild-darwin-stdenv label May 3, 2018

Ericson2314 changed the base branch from master to staging May 3, 2018 20:04

GrahamcOfBorg added 6.topic: erlang 6.topic: fetch 6.topic: golang 6.topic: haskell 6.topic: nixos 6.topic: python labels May 3, 2018

GrahamcOfBorg added 2.status: merge conflict and removed 6.topic: haskell 6.topic: nixos 6.topic: python 6.topic: qt/kde 6.topic: ruby 6.topic: vim 8.has: changelog 8.has: documentation 8.has: module (update) 2.status: merge conflict labels May 3, 2018

bgamari added 11 commits May 3, 2018 17:04

mmc-utils: Enable cross-compilation

67e4ed9

net-tools: Enable cross-compilation

edd92f9

tcp-wrappers: Enable cross-compilation

10fa4a5

This package, like so many others, assumes that ar is unprefixed.

libnl: Enable parallel building

d8c744b

This is a long enough derivation to make this worthwhile.

runit: Enable cross-compilation

6971c50

curl: Enable cross-compilation

d23f71d

By fixing toolchain paths

rng-tools: Enable cross-compilation

a09aa86

lib.systems.gnu: Accept gnueabi as a gnu platform

8b32cfd

ldns: Enable cross-compilation

860c81d

glib: Enable cross-compilation

da48154

glib needs gettext for the build machine. Since gettext is also a propagatedBuildInput I'm assuming that it's also used at runtime.

avahi: Enable cross-compilation

86a778a

Ericson2314 force-pushed the cross-fixes-1 branch from ea49097 to 86a778a Compare May 3, 2018 21:06

GrahamcOfBorg removed the 2.status: merge conflict label May 3, 2018

Ericson2314 merged commit 8ac79d2 into NixOS:staging May 3, 2018

dtzWill reviewed May 3, 2018

View reviewed changes

A selection of cross-compiling fixes #39420

A selection of cross-compiling fixes #39420

Conversation

bgamari commented Apr 24, 2018 • edited

Motivation for this change

Things done

bgamari commented Apr 24, 2018

dtzWill commented Apr 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtzWill commented Apr 24, 2018

bgamari commented Apr 30, 2018 • edited

Ericson2314 commented May 1, 2018

bgamari commented May 3, 2018

Ericson2314 commented May 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgamari commented May 3, 2018

dtzWill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented May 3, 2018

bgamari commented May 4, 2018

bgamari commented Apr 24, 2018 •

edited

bgamari commented Apr 30, 2018 •

edited