stdenv: cleanup darwin bootstrapping #43561

LnL7 · 2018-07-15T08:45:24Z

Motivation for this change

I finally spent some time to clean this up. 😄
Also gets rid of the full python and some of it's dependencies in the stdenv build closure.

db, gdbm, openssl, readline, sqlite and python2 no longer cause an stdenv rebuild.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

LnL7 · 2018-07-15T08:48:09Z

pkgs/stdenv/darwin/default.nix

-      # Hack to make sure we don't link ncurses in bootstrap tools. The proper
-      # solution is to avoid passing -L/nix-store/...-bootstrap-tools/lib,
-      # quite a sledgehammer just to get the C runtime.
-      gettext = super.gettext.overrideAttrs (old: {


@Ericson2314 why was it composed like this?

You mean not part of persistent? I gather to separate concerns: we aren't "persisting" anything with this override.

Did I do this? I kind of like it the way it was, now that I figure out what's going on :)

GrahamcOfBorg · 2018-07-15T09:05:52Z

Success on aarch64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

copying path '/nix/store/pvi2gcj6ld1110hq50jvvpgbx5vqqlsf-gnutar-1.30' from 'https://cache.nixos.org'...
copying path '/nix/store/md8970cv4wx2f372phrchv2kkpncy3b4-glibc-2.27-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/67nqds5kzci6cdd83l333m978kj0vjsa-patchelf-0.9' from 'https://cache.nixos.org'...
copying path '/nix/store/6klshpgrh4pzvj4vry2gghcg3fr7v38c-gcc-7.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/sbg5zpv22yx42ybams9lhzkl6338gsa3-diffutils-3.6' from 'https://cache.nixos.org'...
copying path '/nix/store/wy50l2hng1qgz9hcn9al5phbzpzkgkyx-findutils-4.6.0' from 'https://cache.nixos.org'...
copying path '/nix/store/92l7k9z192gicq286mfdsv00h765w5ci-binutils-wrapper-2.30' from 'https://cache.nixos.org'...
copying path '/nix/store/py7iw171q6zigwl9vf25ac7d14n6visx-gcc-wrapper-7.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/wkxcvf9babisn0hvwl51krkq3p042088-stdenv-linux' from 'https://cache.nixos.org'...
/nix/store/wkxcvf9babisn0hvwl51krkq3p042088-stdenv-linux

GrahamcOfBorg · 2018-07-15T09:06:26Z

Success on x86_64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

/nix/store/pvnkb8ppsg71h2z95vza5fff4m98l7gi-stdenv-linux

FRidh · 2018-07-15T09:16:38Z

pkgs/development/interpreters/python/cpython/2.7/boot.nix

@@ -26,7 +26,7 @@ stdenv.mkDerivation rec {
  LDFLAGS = optionalString (!stdenv.isDarwin) "-lgcc_s";
  NIX_CFLAGS_COMPILE = optionalString stdenv.isDarwin "-msse2";

-  buildInputs = optionals stdenv.isDarwin [ CF configd ];


What were these for? I think the other interpreters still have these.

Not sure about configd but that's probably optional since it was already disabled. As for CF, the darwin bootstrap-tools include CoreFoundation so that's used while building the stdenv itself.

So we can probably remove them from the other cpythons? I bet it was just copy pasted from this one - someone not realizing that only 2.7 is used in bootstrapping.

Please do it in a separate commit, but within this PR.

No, we can't make packages from the final set depend on bootstrap tools. I'll add a comment to the expression explaining what's going on.

Okay, it might be clearer to leave these in but just null them out in stdenv/darwin/default.nix

I am still a little confused on why this is needed. Packages don't usually depend on CF or configd. Doesn't the stdenv provide a version that non-boot cpython can pick up?

GrahamcOfBorg · 2018-07-15T09:39:19Z

Failure on x86_64-darwin (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/ksli2gy5754vghw5yxvr9xadyykgmysf-cctools-port-895.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/94mc2w0f2rckrra8yv9qlks0r0whx3zs-hook.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/rwk5ysfscqmvx7vgn63qvxgzchcn2ngq-ICU-osx-10.10.5.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/hdb410czybrizzhk1bir4didiik0jrk9-cctools-binutils-darwin.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/4247sjj27d99farlqj2xn825mdzxxib2-gnutar-1.30.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/7wwkil47mci10643vpq0q4vh1bk86b47-CF-osx-10.10.5.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/r8wf4ykc3ayzgsnjmrx1lncqx2h9f6xw-cctools-binutils-darwin-wrapper.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/50k0av854hw9wmcyp0ydx0f2h4129ib7-clang-wrapper-5.0.2.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/pl96777bczlhxf6p58qzf3psdyi9r83s-stdenv-darwin.drv': 36 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/pl96777bczlhxf6p58qzf3psdyi9r83s-stdenv-darwin.drv' failed

matthewbauer · 2018-07-16T16:51:05Z

pkgs/stdenv/darwin/default.nix

@@ -188,7 +188,15 @@ in rec {
  };

  stage1 = prevStage: let
-    persistent = _: super: { python = super.python.override { configd = null; }; };
+    persistent = self: super: with prevStage; {
+      cmake = super.cmake.override {


Could you verify that this still works on cross? I can't remember the exact reason but we had to do super.buildPackages.cmake to avoid cross compilation issues.

I don't fully understand what buildPackages implies in this context.

This is fine @matthewbauer because the bootstrapping is explicitly native. We don't need to worry about persisting things with a target platform in the final stage, as if the host platform changes stdenv overrides are already chucked out. cmake has no target platform, and this is also an early stage platform, so we are good.

Ericson2314 · 2018-07-16T22:13:40Z

pkgs/stdenv/darwin/default.nix

@@ -281,6 +292,15 @@ in rec {
        ncurses libffi zlib gmp pcre gnugrep
        coreutils findutils diffutils patchutils;

+      # Hack to make sure we don't link ncurses in bootstrap tools. The proper
+      # solution is to avoid passing -L/nix-store/...-bootstrap-tools/lib,
+      # quite a sledgehammer just to get the C runtime.


Yes eventually let's change bootstrap tools to keep builds separate rather than going FHS for no reason whatsoever!

Also gets rid of the full python and some of it's dependencies in the stdenv build closure.

GrahamcOfBorg · 2018-07-21T15:04:59Z

Success on aarch64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

/nix/store/wkxcvf9babisn0hvwl51krkq3p042088-stdenv-linux

GrahamcOfBorg · 2018-07-21T15:05:02Z

Success on x86_64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

/nix/store/pvnkb8ppsg71h2z95vza5fff4m98l7gi-stdenv-linux

GrahamcOfBorg · 2018-07-21T18:59:51Z

Failure on x86_64-darwin (full log)

Attempted: stdenv

Partial log (click to expand)

make[1]: Leaving directory '/private/tmp/nix-build-gnutar-1.30.drv-0/tar-1.30'
post-installation fixup
gzipping man pages under /nix/store/a9mva9v73fpdf93n2id1ayi32jz7l68x-gnutar-1.30/share/man/
strip is /nix/store/4q1c84f1ynfvsd85a932375sg0jc89v8-bootstrap-tools/bin/strip
stripping (with command strip and flags -S) in /nix/store/a9mva9v73fpdf93n2id1ayi32jz7l68x-gnutar-1.30/bin
patching script interpreter paths in /nix/store/a9mva9v73fpdf93n2id1ayi32jz7l68x-gnutar-1.30
strip is /nix/store/4q1c84f1ynfvsd85a932375sg0jc89v8-bootstrap-tools/bin/strip
patching script interpreter paths in /nix/store/dy6q32qi8zsqkima952f71y5syawn5dr-gnutar-1.30-info
cannot build derivation '/nix/store/vrpijb7qxp3w0iyf8k94yd03nkllv5hr-stdenv-darwin.drv': 5 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/vrpijb7qxp3w0iyf8k94yd03nkllv5hr-stdenv-darwin.drv' failed

matthewbauer · 2018-07-21T20:52:30Z

Great work!

LnL7 requested a review from FRidh as a code owner July 15, 2018 08:45

GrahamcOfBorg added the 6.topic: darwin Running or building packages on Darwin label Jul 15, 2018

LnL7 changed the base branch from master to staging July 15, 2018 08:46

LnL7 force-pushed the darwin-llvm-boot branch from 504d4d8 to b527098 Compare July 15, 2018 08:47

LnL7 commented Jul 15, 2018

View reviewed changes

GrahamcOfBorg added 6.topic: python 6.topic: stdenv Standard environment 10.rebuild-darwin-stdenv 10.rebuild-darwin: 501+ 10.rebuild-linux: 0 labels Jul 15, 2018

FRidh reviewed Jul 15, 2018

View reviewed changes

matthewbauer reviewed Jul 16, 2018

View reviewed changes

Ericson2314 reviewed Jul 16, 2018

View reviewed changes

stdenv: cleanup darwin bootstrapping

93c34c7

Also gets rid of the full python and some of it's dependencies in the stdenv build closure.

LnL7 force-pushed the darwin-llvm-boot branch from b527098 to 93c34c7 Compare July 21, 2018 14:51

LnL7 merged commit d3cc05e into NixOS:staging Jul 21, 2018

LnL7 deleted the darwin-llvm-boot branch July 21, 2018 20:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stdenv: cleanup darwin bootstrapping #43561

stdenv: cleanup darwin bootstrapping #43561

LnL7 commented Jul 15, 2018

LnL7 Jul 15, 2018 •

edited

Ericson2314 Jul 16, 2018

Ericson2314 Jul 16, 2018 •

edited

GrahamcOfBorg commented Jul 15, 2018

GrahamcOfBorg commented Jul 15, 2018

FRidh Jul 15, 2018

LnL7 Jul 15, 2018 •

edited

matthewbauer Jul 16, 2018

FRidh Jul 16, 2018

LnL7 Jul 16, 2018 •

edited

matthewbauer Jul 16, 2018

matthewbauer Jul 16, 2018

GrahamcOfBorg commented Jul 15, 2018

matthewbauer Jul 16, 2018

LnL7 Jul 16, 2018

Ericson2314 Jul 16, 2018

Ericson2314 Jul 16, 2018

GrahamcOfBorg commented Jul 21, 2018

GrahamcOfBorg commented Jul 21, 2018

GrahamcOfBorg commented Jul 21, 2018

matthewbauer commented Jul 21, 2018

stdenv: cleanup darwin bootstrapping #43561

stdenv: cleanup darwin bootstrapping #43561

Conversation

LnL7 commented Jul 15, 2018

Motivation for this change

Things done

LnL7 Jul 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Jul 16, 2018 • edited

Choose a reason for hiding this comment

GrahamcOfBorg commented Jul 15, 2018

GrahamcOfBorg commented Jul 15, 2018

Choose a reason for hiding this comment

LnL7 Jul 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LnL7 Jul 16, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Jul 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Jul 21, 2018

GrahamcOfBorg commented Jul 21, 2018

GrahamcOfBorg commented Jul 21, 2018

matthewbauer commented Jul 21, 2018

LnL7 Jul 15, 2018 •

edited

Ericson2314 Jul 16, 2018 •

edited

LnL7 Jul 15, 2018 •

edited

LnL7 Jul 16, 2018 •

edited