-
-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
misc pkgs: Darwin -> Linux cross compilation #46534
misc pkgs: Darwin -> Linux cross compilation #46534
Conversation
pkgs/top-level/all-packages.nix
Outdated
@@ -9513,6 +9513,12 @@ with pkgs; | |||
installLocales = config.glibc.locales or false; | |||
}; | |||
|
|||
# Provided by libc on Operating Systems that use the Extensible Linker Format. | |||
elf-header = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you seen if libelf
above would work? Not sure how that works currently but it might be a little bit more generic of the name. elf-header IMO is a little two awkward. No package will ever list a dependency on "elf-header" but may for libelf.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
libelf
is something else. I bet paxctl could be rewritten against it, but it's not a drop-in replacement. (libelf
also looks more like the darwin DTrace header.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm... I guess it's okay then. It's just semantically kind of ugly to me.
pkgs/stdenv/cross/default.nix
Outdated
(hostPlatform.isLinux && !buildPlatform.isLinux) | ||
[ buildPackages.patchelf buildPackages.paxctl ] | ||
++ lib.optional | ||
(let f = p: !p.isx86 || p.libc == "musl"; in f hostPlatform && !(f buildPlatform)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems a little bit too much. I think I would prefer the more verbose: (!hostPlatform.isx86 || hostPlatform.libc == "musl" && !(!buildPlatform.isx86 || buildPlatform.libc == "musl")
. Would it hurt to just do this everywhere? It's just a hook so maybe it's smart enough to avoid the conditional completely?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do want to do this everywhere, but unfortunately I don't see a good way to deduplicate it between stdenv/linux
and stdenv/cross
since stdenv/generic
doesn't get passed whole package sets.
Or...we could just use one of the builtin fetchers to get the new scripts so we don't even need most of nixpkgs 😈.
Great work!!! When I looked at this in June I got stuck on the case insensitive glibc. Nice that there is a patch for that. It would be good to report this to glibc. I suspect they already know about it but just maybe giving them the patch would make it a priority (most likely they will not apply it immediately though). This is my old branch: https://github.com/matthewbauer/nixpkgs/commits/cross Everything is hopefully upstreamed. |
@@ -96,7 +94,8 @@ callPackage ./common.nix { inherit stdenv; } { | |||
mv $bin/bin/getconf_ $bin/bin/getconf | |||
''; | |||
|
|||
separateDebugInfo = true; | |||
# Hack to get around eval issue. | |||
separateDebugInfo = !stdenv.isDarwin; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be useful to do in mk-derivation. I don't think separateDebugInfo has ever worked in macOS.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah if we remove the assert in mkDerivation
and instead make it part of meta.broken
or something, that would give us the required laziness.
makeFlags = [ | ||
"SHELL=bash" | ||
# Avoid use of runtime build->host compilers for checks | ||
"cc-version:=9999" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ugh... Another thing to report upstream, but it is probably fine for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well... it's arguably not a bug. I checked and this version is just used to work around bugs in old GCCs; I assume/hope newer GCCs are given better testing so hopefully it won't a happen again. But we could generate the actual version number thing just as a precaution.
Then again, upstream could provide the right hook in the configure script so faking this number wouldn't be downstream's burden.
Really great kudos from my side. I could only skim it very fast. However, i will try it tommorow on couple of our docker base images which are using dockerTools and are built with |
pkgs/development/libraries/glibc/0001-Fix-build-on-case-insensative-filesystems.patch
Outdated
Show resolved
Hide resolved
How much were you able to cross compile? I would think some things could still go wrong when cross compiling a NixOS image. |
@matthewbauer Yeah I just did stdenv and random easy stuff like |
4a7751c
to
bab2b93
Compare
https://sourceware.org/bugzilla/show_bug.cgi?id=72 Uhoh, upstream so far hasn't been interested. |
Someone sent an article recently that might be useful: https://worthdoingbadly.com/casesensitive-iossim/ Basically you can do EDIT: Just now realized it requires root privileges & a PID. Might be something worth doing in Nix though (which should know the PID). |
bab2b93
to
54047e5
Compare
Holy shit @matthewbauer I had no idea that was a thing. Can you file that as a ticket somewhere so we don't forget it with all the stuff going on? I'd love to get that integrated into Nix |
Oh right! I remember seeing that somewhere now. |
This PR illustrates my concerns with cross-compilation support: it adds dozens of patches, special casing, extra build flags, pseudo-packages like |
@edolstra I understand the constrains and the extra maintenance burden for our derivation, however i have to admit that building docker images on macOS is one of my daily jobs in developer land. Our developers at holidaycheck have to run a remote builder e.g. lnl7/nix-docker to use dockerTools daily. Latter has limited resources on macOS with Docker4Mac. I would like to ditch DockerForMac if we could get binary cached cross compiled glibc and gcc packages. Thus dockerTools will be closer to nix and lesser dependent on docker4mac. Developers writing jvm/nodejs/haskell services at our place really have more concerns than maintaining docker4mac's glitches just to build images. I admit most of the time those images are built through a linux-baked pipeline, but people writing dockerTools-Expression need to test them upfront. |
In case it isn't clear already, this is far and a way the most popular thing my work enables. |
#46574 Contains some non-mass-rebuild portions of this |
89e5e5e
to
bf6e64c
Compare
Carefully fake cc-version and cc-fullversion to avoid needing a compiler for the kernel itself to build the headers. For some reason, doing `make install_headers` twice, first without INSTALL_HDR_PATH=$out then with, is neccessary to get this to work.
bf6e64c
to
f6456c3
Compare
f6456c3
to
bbd12f5
Compare
FWIW after I removed the conditionals this broke stdenv bootstrap by
causing the final stdenv to reference bootstrap tools.
More investigation needed, but please be sure to check this isn't broken
whenever the conditionals are dropped.
…On Wed, 24 Oct 2018 07:37:04 -0700, John Ericson ***@***.***> wrote:
Merged #46534 into master.
--
You are receiving this because your review was requested.
Reply to this email directly or view it on GitHub:
#46534 (comment) part: text/html
|
Linux bootstrap? Weird and bummer. I'll open the clean up PR tomorrow so we have something to investigate. |
Great, tag me on it if you would please :).
I just wanted to draw attention towards it so it wasn't overlooked.
Some bootstrap issues --although not sure this one-- have been
surprisingly buried and only noticed long after the breaking commits
when upgrading (or using an alternative :P) libc or so.
As a result I may be a bit overly concerned but better safe
than sorry and easy enough to check if you know to do so :).
There are enough factors in my environment that it's possible
it's due to some other complication (maybe even a simple typo dropping
the conditionals on my part!) but instead of worrying about that
let's just proceed and check things are good :).
(as you suggest)
Thanks, have a good one!
…On Wed, 24 Oct 2018 18:58:34 -0700, John Ericson ***@***.***> wrote:
Linux bootstrap? Weird and bummer. I'll open the clean up PR tomorrow so we have something to investigate.
--
You are receiving this because your review was requested.
Reply to this email directly or view it on GitHub:
#46534 (comment) part: text/html
|
@Ericson2314 Is the following error a glibc issue or related to the builder:
i was trying to build openjdk with cross-compilation after the PR merge. |
Yeah that's a glibc faillure. Try a |
On Thu, 25 Oct 2018 12:32:24 -0700, John Ericson ***@***.***> wrote:
Yeah that's a glibc faillure. Try a `crossSystem` of `{ config = "x86_64-unknown-linux-musl"; }`.
😁 I love just seeing that mentioned, it's so cool it's even a
little bit possible to so easily express that.
Less awesome is this: unfortunately I haven't yet added support for
building openjdk w/musl, so presently attempting to build it will
assert or throw an error :(.
There's no reason it would be overly difficult,
and I'd be happy to provide guidance if anyone wanted to tackle it.
… --
You are receiving this because your review was requested.
Reply to this email directly or view it on GitHub:
#46534 (comment) part: text/html
|
#49129 is the staging cleanup with the fix. |
Motivation for this change
Carefully changed to not be a mass rebuild, I'll clean up after (or @oxij can call me out :)). Glibc patches delayed temporarily, but musl works fine.
Fixes #27934
Things done
sandbox
innix.conf
on non-NixOS)nix-shell -p nox --run "nox-review wip"
./result/bin/
)nix path-info -S
before and after)CC @3noch @periklis