misc pkgs: Darwin -> Linux cross compilation #46534

Ericson2314 · 2018-09-11T18:08:00Z

Motivation for this change

Carefully changed to not be a mass rebuild, I'll clean up after (or @oxij can call me out :)). Glibc patches delayed temporarily, but musl works fine.

Fixes #27934

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

CC @3noch @periklis

matthewbauer · 2018-09-11T18:13:20Z

pkgs/top-level/all-packages.nix

@@ -9513,6 +9513,12 @@ with pkgs;
    installLocales = config.glibc.locales or false;
  };

+  # Provided by libc on Operating Systems that use the Extensible Linker Format.
+  elf-header =


Have you seen if libelf above would work? Not sure how that works currently but it might be a little bit more generic of the name. elf-header IMO is a little two awkward. No package will ever list a dependency on "elf-header" but may for libelf.

libelf is something else. I bet paxctl could be rewritten against it, but it's not a drop-in replacement. (libelf also looks more like the darwin DTrace header.)

Hmm... I guess it's okay then. It's just semantically kind of ugly to me.

matthewbauer · 2018-09-11T18:16:41Z

pkgs/stdenv/cross/default.nix

+             (hostPlatform.isLinux && !buildPlatform.isLinux)
+             [ buildPackages.patchelf buildPackages.paxctl ]
+        ++ lib.optional
+             (let f = p: !p.isx86 || p.libc == "musl"; in f hostPlatform && !(f buildPlatform))


This seems a little bit too much. I think I would prefer the more verbose: (!hostPlatform.isx86 || hostPlatform.libc == "musl" && !(!buildPlatform.isx86 || buildPlatform.libc == "musl"). Would it hurt to just do this everywhere? It's just a hook so maybe it's smart enough to avoid the conditional completely?

I do want to do this everywhere, but unfortunately I don't see a good way to deduplicate it between stdenv/linux and stdenv/cross since stdenv/generic doesn't get passed whole package sets.

Or...we could just use one of the builtin fetchers to get the new scripts so we don't even need most of nixpkgs 😈.

matthewbauer · 2018-09-11T18:18:58Z

Great work!!! When I looked at this in June I got stuck on the case insensitive glibc. Nice that there is a patch for that. It would be good to report this to glibc. I suspect they already know about it but just maybe giving them the patch would make it a priority (most likely they will not apply it immediately though).

This is my old branch:

https://github.com/matthewbauer/nixpkgs/commits/cross

Everything is hopefully upstreamed.

pkgs/development/libraries/glibc/common.nix

matthewbauer · 2018-09-11T18:22:56Z

pkgs/development/libraries/glibc/default.nix

@@ -96,7 +94,8 @@ callPackage ./common.nix { inherit stdenv; } {
      mv $bin/bin/getconf_ $bin/bin/getconf
    '';

-    separateDebugInfo = true;
+    # Hack to get around eval issue.
+    separateDebugInfo = !stdenv.isDarwin;


This might be useful to do in mk-derivation. I don't think separateDebugInfo has ever worked in macOS.

Yeah if we remove the assert in mkDerivation and instead make it part of meta.broken or something, that would give us the required laziness.

matthewbauer · 2018-09-11T18:24:01Z

pkgs/os-specific/linux/kernel-headers/default.nix

+    makeFlags = [
+      "SHELL=bash"
+      # Avoid use of runtime build->host compilers for checks
+      "cc-version:=9999"


ugh... Another thing to report upstream, but it is probably fine for now.

Well... it's arguably not a bug. I checked and this version is just used to work around bugs in old GCCs; I assume/hope newer GCCs are given better testing so hopefully it won't a happen again. But we could generate the actual version number thing just as a precaution.

Then again, upstream could provide the right hook in the configure script so faking this number wouldn't be downstream's burden.

periklis · 2018-09-11T18:36:31Z

Really great kudos from my side. I could only skim it very fast. However, i will try it tommorow on couple of our docker base images which are using dockerTools and are built with lnl7/nix-docker. This will enhance our build chain a lot at Holidaycheck.

pkgs/development/libraries/glibc/0001-Fix-build-on-case-insensative-filesystems.patch

matthewbauer · 2018-09-11T19:07:19Z

How much were you able to cross compile? I would think some things could still go wrong when cross compiling a NixOS image.

Ericson2314 · 2018-09-11T19:14:31Z

@matthewbauer Yeah I just did stdenv and random easy stuff like hello and ed. The infrastructure works, but I make no claim that arbitrary things will build on case-insensitive filesystems :D.

Ericson2314 · 2018-09-11T20:12:24Z

https://sourceware.org/bugzilla/show_bug.cgi?id=72 Uhoh, upstream so far hasn't been interested.

matthewbauer · 2018-09-11T20:13:18Z

Someone sent an article recently that might be useful:

https://worthdoingbadly.com/casesensitive-iossim/

Basically you can do taskpolicy -X to enable case sensitivity in macOS. taskpolicy is part of system_cmds but not currently built by us. We can probably fix that easily though.

EDIT: Just now realized it requires root privileges & a PID. Might be something worth doing in Nix though (which should know the PID).

copumpkin · 2018-09-11T21:44:29Z

Holy shit @matthewbauer I had no idea that was a thing. Can you file that as a ticket somewhere so we don't forget it with all the stuff going on? I'd love to get that integrated into Nix

LnL7 · 2018-09-11T21:47:13Z

Oh right! I remember seeing that somewhere now.

edolstra · 2018-09-12T09:45:23Z

This PR illustrates my concerns with cross-compilation support: it adds dozens of patches, special casing, extra build flags, pseudo-packages like elf-header etc., all to support a relatively obscure use case (namely macOS -> Linux cross-compilation). I mean, don't most macOS users just use VMs to build Linux stuff?

periklis · 2018-09-12T10:47:50Z

@edolstra I understand the constrains and the extra maintenance burden for our derivation, however i have to admit that building docker images on macOS is one of my daily jobs in developer land. Our developers at holidaycheck have to run a remote builder e.g. lnl7/nix-docker to use dockerTools daily. Latter has limited resources on macOS with Docker4Mac. I would like to ditch DockerForMac if we could get binary cached cross compiled glibc and gcc packages. Thus dockerTools will be closer to nix and lesser dependent on docker4mac. Developers writing jvm/nodejs/haskell services at our place really have more concerns than maintaining docker4mac's glitches just to build images. I admit most of the time those images are built through a linux-baked pipeline, but people writing dockerTools-Expression need to test them upfront.

Ericson2314 · 2018-09-13T02:14:47Z

@edolstra

I mean, don't most macOS users just use VMs to build Linux stuff?

In case it isn't clear already, this is far and a way the most popular thing my work enables.

Ericson2314 · 2018-09-13T02:15:17Z

#46574 Contains some non-mass-rebuild portions of this

Carefully fake cc-version and cc-fullversion to avoid needing a compiler for the kernel itself to build the headers. For some reason, doing `make install_headers` twice, first without INSTALL_HDR_PATH=$out then with, is neccessary to get this to work.

…ross

dtzWill · 2018-10-24T20:33:02Z

FWIW after I removed the conditionals this broke stdenv bootstrap by causing the final stdenv to reference bootstrap tools. More investigation needed, but please be sure to check this isn't broken whenever the conditionals are dropped.

…

On Wed, 24 Oct 2018 07:37:04 -0700, John Ericson ***@***.***> wrote: Merged #46534 into master. -- You are receiving this because your review was requested. Reply to this email directly or view it on GitHub: #46534 (comment) part: text/html

Ericson2314 · 2018-10-25T01:58:29Z

Linux bootstrap? Weird and bummer. I'll open the clean up PR tomorrow so we have something to investigate.

dtzWill · 2018-10-25T03:17:48Z

Great, tag me on it if you would please :). I just wanted to draw attention towards it so it wasn't overlooked. Some bootstrap issues --although not sure this one-- have been surprisingly buried and only noticed long after the breaking commits when upgrading (or using an alternative :P) libc or so. As a result I may be a bit overly concerned but better safe than sorry and easy enough to check if you know to do so :). There are enough factors in my environment that it's possible it's due to some other complication (maybe even a simple typo dropping the conditionals on my part!) but instead of worrying about that let's just proceed and check things are good :). (as you suggest) Thanks, have a good one!

…

On Wed, 24 Oct 2018 18:58:34 -0700, John Ericson ***@***.***> wrote: Linux bootstrap? Weird and bummer. I'll open the clean up PR tomorrow so we have something to investigate. -- You are receiving this because your review was requested. Reply to this email directly or view it on GitHub: #46534 (comment) part: text/html

periklis · 2018-10-25T10:22:08Z

@Ericson2314 Is the following error a glibc issue or related to the builder:

[Run cross-compilation jobset] builder for '/nix/store/8fcx14mfnlzm03704dwy03ldnsjgy490-glibc-2.27-x86_64-linux-gnu.drv' failed with exit code 1; last 10 log lines:
[Run cross-compilation jobset]   rm -f /private/var/folders/lg/gsg812dx2llcqvrxfchsngn80000gp/T/nix-build-glibc-2.27-x86_64-linux-gnu.drv-0/build/localedata/stamp.os; > /private/var/folders/lg/gsg812dx2llcqvrxfchsngn80000gp/T/nix-build-glibc-2.27-x86_64-linux-gnu.drv-0/build/localedata/stamp.os
[Run cross-compilation jobset]   make[2]: Leaving directory '/private/var/folders/lg/gsg812dx2llcqvrxfchsngn80000gp/T/nix-build-glibc-2.27-x86_64-linux-gnu.drv-0/glibc-2.27/localedata'
[Run cross-compilation jobset]   make  subdir=iconvdata -C iconvdata ..=../ subdir_lib
[Run cross-compilation jobset]   make[2]: Entering directory '/private/var/folders/lg/gsg812dx2llcqvrxfchsngn80000gp/T/nix-build-glibc-2.27-x86_64-linux-gnu.drv-0/glibc-2.27/iconvdata'
[Run cross-compilation jobset]   extra-module.mk:11: *** Too many open files.  Stop.
[Run cross-compilation jobset]   make[2]: Leaving directory '/private/var/folders/lg/gsg812dx2llcqvrxfchsngn80000gp/T/nix-build-glibc-2.27-x86_64-linux-gnu.drv-0/glibc-2.27/iconvdata'
[Run cross-compilation jobset]   make[1]: *** [Makefile:215: iconvdata/subdir_lib] Error 2
[Run cross-compilation jobset]   make[1]: Leaving directory '/private/var/folders/lg/gsg812dx2llcqvrxfchsngn80000gp/T/nix-build-glibc-2.27-x86_64-linux-gnu.drv-0/glibc-2.27'
[Run cross-compilation jobset]   make: *** [Makefile:9: all] Error 2
[Run cross-compilation jobset]   builder for '/nix/store/8fcx14mfnlzm03704dwy03ldnsjgy490-glibc-2.27-x86_64-linux-gnu.drv' failed with exit code 2

i was trying to build openjdk with cross-compilation after the PR merge.

Ericson2314 · 2018-10-25T19:32:20Z

Yeah that's a glibc faillure. Try a crossSystem of { config = "x86_64-unknown-linux-musl"; }.

dtzWill · 2018-10-25T22:19:24Z

On Thu, 25 Oct 2018 12:32:24 -0700, John Ericson ***@***.***> wrote: Yeah that's a glibc faillure. Try a `crossSystem` of `{ config = "x86_64-unknown-linux-musl"; }`.

😁 I love just seeing that mentioned, it's so cool it's even a little bit possible to so easily express that. Less awesome is this: unfortunately I haven't yet added support for building openjdk w/musl, so presently attempting to build it will assert or throw an error :(. There's no reason it would be overly difficult, and I'd be happy to provide guidance if anyone wanted to tackle it.

…

-- You are receiving this because your review was requested. Reply to this email directly or view it on GitHub: #46534 (comment) part: text/html

Ericson2314 · 2018-10-26T01:47:17Z

#49129 is the staging cleanup with the fix.

Ericson2314 requested review from matthewbauer, LnL7 and dtzWill September 11, 2018 18:08

GrahamcOfBorg added 6.topic: darwin Running or building packages on Darwin 6.topic: stdenv Standard environment labels Sep 11, 2018

matthewbauer reviewed Sep 11, 2018

View reviewed changes

pkgs/development/libraries/glibc/common.nix Outdated Show resolved Hide resolved

matthewbauer reviewed Sep 11, 2018

View reviewed changes

LnL7 reviewed Sep 11, 2018

View reviewed changes

pkgs/development/libraries/glibc/0001-Fix-build-on-case-insensative-filesystems.patch Outdated Show resolved Hide resolved

Ericson2314 force-pushed the darwin-to-linux-cross branch from 4a7751c to bab2b93 Compare September 11, 2018 19:55

matthewbauer requested a review from copumpkin September 11, 2018 20:15

Ericson2314 force-pushed the darwin-to-linux-cross branch from bab2b93 to 54047e5 Compare September 11, 2018 20:53

Ericson2314 added 6.topic: cross-compilation Building packages on a different sort platform than than they will be run on 6.topic: portability General portability concerns, not specific to cross-compilation or a specific platform labels Sep 11, 2018

Ericson2314 mentioned this pull request Sep 11, 2018

glibc: Use lib.enableFeature and friends #46544

Merged

9 tasks

matthewbauer mentioned this pull request Sep 11, 2018

Set macOS/kernel to be case sensitive in builder NixOS/nix#2415

Open

Ericson2314 force-pushed the darwin-to-linux-cross branch from 89e5e5e to bf6e64c Compare October 24, 2018 02:06

Ericson2314 changed the base branch from staging to master October 24, 2018 02:10

LnL7 and others added 2 commits October 23, 2018 23:20

release-cross: Cross compile from all 3 supported platforms

22df942

Ericson2314 mentioned this pull request Oct 24, 2018

misc pkgs: Darwin -> Linux cross compilation for 18.09 #48931

Merged

9 tasks

Ericson2314 force-pushed the darwin-to-linux-cross branch from bf6e64c to f6456c3 Compare October 24, 2018 03:34

Merge remote-tracking branch 'upstream/master' into darwin-to-linux-c…

bbd12f5

…ross

Ericson2314 force-pushed the darwin-to-linux-cross branch from f6456c3 to bbd12f5 Compare October 24, 2018 13:31

Ericson2314 merged commit 7d96d15 into NixOS:master Oct 24, 2018

Ericson2314 deleted the darwin-to-linux-cross branch October 24, 2018 14:36

maxmcd mentioned this pull request Oct 21, 2020

Implement the nixos darwin case insensitivity hack maxmcd/bramble#8

Closed

uri-canva mentioned this pull request Aug 4, 2021

dockerTools.buildImage on macOS #132492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

misc pkgs: Darwin -> Linux cross compilation #46534

misc pkgs: Darwin -> Linux cross compilation #46534

Ericson2314 commented Sep 11, 2018 •

edited

matthewbauer Sep 11, 2018

Ericson2314 Sep 11, 2018

matthewbauer Sep 11, 2018

matthewbauer Sep 11, 2018

Ericson2314 Sep 11, 2018 •

edited

matthewbauer commented Sep 11, 2018 •

edited

matthewbauer Sep 11, 2018

Ericson2314 Sep 11, 2018

matthewbauer Sep 11, 2018

Ericson2314 Sep 11, 2018

periklis commented Sep 11, 2018

matthewbauer commented Sep 11, 2018

Ericson2314 commented Sep 11, 2018

Ericson2314 commented Sep 11, 2018

matthewbauer commented Sep 11, 2018 •

edited

copumpkin commented Sep 11, 2018

LnL7 commented Sep 11, 2018

edolstra commented Sep 12, 2018

periklis commented Sep 12, 2018

Ericson2314 commented Sep 13, 2018

Ericson2314 commented Sep 13, 2018

dtzWill commented Oct 24, 2018 via email

Ericson2314 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

periklis commented Oct 25, 2018

Ericson2314 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

Ericson2314 commented Oct 26, 2018

misc pkgs: Darwin -> Linux cross compilation #46534

misc pkgs: Darwin -> Linux cross compilation #46534

Conversation

Ericson2314 commented Sep 11, 2018 • edited

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Sep 11, 2018 • edited

Choose a reason for hiding this comment

matthewbauer commented Sep 11, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

periklis commented Sep 11, 2018

matthewbauer commented Sep 11, 2018

Ericson2314 commented Sep 11, 2018

Ericson2314 commented Sep 11, 2018

matthewbauer commented Sep 11, 2018 • edited

copumpkin commented Sep 11, 2018

LnL7 commented Sep 11, 2018

edolstra commented Sep 12, 2018

periklis commented Sep 12, 2018

Ericson2314 commented Sep 13, 2018

Ericson2314 commented Sep 13, 2018

dtzWill commented Oct 24, 2018 via email

Ericson2314 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

periklis commented Oct 25, 2018

Ericson2314 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

Ericson2314 commented Oct 26, 2018

Ericson2314 commented Sep 11, 2018 •

edited

Ericson2314 Sep 11, 2018 •

edited

matthewbauer commented Sep 11, 2018 •

edited

matthewbauer commented Sep 11, 2018 •

edited