treewide: mass rebuild complicated cleanups #48721

oxij · 2018-10-19T18:53:47Z

Motivation for this change

Continuation of #47233.

The first two patches homogenize gcc expressions (i.e. make diff between them as small as possible) in the hope that they could be merged into a single expression eventually.

I have no idea whether the third patch does the right thing.

Things done

Running on top of this for two months without issues.

/cc @Ericson2314

Looks scary but it is a noop.

GrahamcOfBorg · 2018-10-19T19:06:50Z

Success on x86_64-linux (full log)

Attempted: llvm_5, stdenv

Partial log (click to expand)

these paths will be fetched (37.43 MiB download, 216.54 MiB unpacked):
  /nix/store/3drlhmj7fvl8sfa9np6nb8axk28p3419-llvm-5.0.2-lib
  /nix/store/9vy4mcydwrdxsybiwrfb9r1yc5j7m7i8-llvm-5.0.2
copying path '/nix/store/3drlhmj7fvl8sfa9np6nb8axk28p3419-llvm-5.0.2-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/9vy4mcydwrdxsybiwrfb9r1yc5j7m7i8-llvm-5.0.2' from 'https://cache.nixos.org'...
/nix/store/9vy4mcydwrdxsybiwrfb9r1yc5j7m7i8-llvm-5.0.2
/nix/store/9pifap75pkb9ifmdqh0irn8wgc3q38s6-stdenv-linux

GrahamcOfBorg · 2018-10-19T19:19:34Z

Success on aarch64-linux (full log)

Attempted: llvm_5, stdenv

Partial log (click to expand)

/nix/store/26wnsb2s0p7n0dbm6l3q28qd3nwmzjdi-llvm-5.0.2-python/share/opt-viewer/opt-viewer.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/74b1xd5srd6ja54xwznvbrd5hcfl6k1n-python-2.7.15/bin/python2.7"
/nix/store/26wnsb2s0p7n0dbm6l3q28qd3nwmzjdi-llvm-5.0.2-python/share/opt-viewer/opt-stats.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/74b1xd5srd6ja54xwznvbrd5hcfl6k1n-python-2.7.15/bin/python2.7"
/nix/store/26wnsb2s0p7n0dbm6l3q28qd3nwmzjdi-llvm-5.0.2-python/share/opt-viewer/opt-diff.py: interpreter directive changed from "/usr/bin/env python2.7" to "/nix/store/74b1xd5srd6ja54xwznvbrd5hcfl6k1n-python-2.7.15/bin/python2.7"
checking for references to /build in /nix/store/26wnsb2s0p7n0dbm6l3q28qd3nwmzjdi-llvm-5.0.2-python...
shrinking RPATHs of ELF executables and libraries in /nix/store/1zmm9vy864psdnmxmv75nbmd8w9dx9sh-llvm-5.0.2-lib
shrinking /nix/store/1zmm9vy864psdnmxmv75nbmd8w9dx9sh-llvm-5.0.2-lib/lib/libLLVM-5.0.so
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/1zmm9vy864psdnmxmv75nbmd8w9dx9sh-llvm-5.0.2-lib/lib
patching script interpreter paths in /nix/store/1zmm9vy864psdnmxmv75nbmd8w9dx9sh-llvm-5.0.2-lib
checking for references to /build in /nix/store/1zmm9vy864psdnmxmv75nbmd8w9dx9sh-llvm-5.0.2-lib...

Ericson2314 · 2018-10-19T19:23:14Z

Thanks!!

The last commit should just be dropped; everything else is good. The issue is while technically it is correct to set that env var based on the target platform, the vast majority of package are not sensitiveness to the target platform, and so we'd like to keep it that way way. With that uncommented, all build-time deps of cross compiled packages on darwin must be rebuilt, which is a big annoyance.

The solutions I want is @lheckemann's stuff to use absolute dt_needed / sonames, and avoid all this rpath stuff everywhere (on linux and darwin).

oxij · 2018-10-19T19:45:34Z

I see. Done.

GrahamcOfBorg · 2018-10-19T20:00:23Z

Success on x86_64-linux (full log)

Attempted: llvm_5

Partial log (click to expand)

/nix/store/9vy4mcydwrdxsybiwrfb9r1yc5j7m7i8-llvm-5.0.2

GrahamcOfBorg · 2018-10-19T20:00:36Z

Success on aarch64-linux (full log)

Attempted: llvm_5

Partial log (click to expand)

/nix/store/dlm5rcrmhivihcajhjsxjcx88sfr04nk-llvm-5.0.2

veprbl · 2018-10-19T20:01:19Z

pkgs/development/compilers/gcc/4.9/default.nix

@@ -424,7 +424,8 @@ stdenv.mkDerivation ({
    platforms =
      stdenv.lib.platforms.linux ++
      stdenv.lib.platforms.freebsd ++
-      stdenv.lib.platforms.illumos;
+      stdenv.lib.platforms.illumos ++
+      stdenv.lib.platforms.darwin;


IIRC gcc 4.8 is broken on darwin. Perhaps, the same is true for 4.9. Should probably use the meta.broken attribute to indicate that.

veprbl · 2018-10-19T20:04:43Z

pkgs/development/compilers/gcc/5/default.nix

@@ -240,7 +240,6 @@ stdenv.mkDerivation ({
    # The builder relies on GNU sed (for instance, Darwin's `sed' fails with
    # "-i may not be used with stdin"), and `stdenvNative' doesn't provide it.
    ++ (optional hostPlatform.isDarwin gnused)
-    ++ (optional hostPlatform.isDarwin targetPackages.stdenv.cc.bintools)


I think you missed one on line 238 here.

FRidh · 2018-10-21T06:39:51Z

Waiting for more feedback before merging into staging.

oxij · 2018-10-21T18:35:03Z

@veprbl

I think you missed one on line 238 here.

Oops, right, fixed.

IIRC gcc 4.8 is broken on darwin. Perhaps, the same is true for 4.9. Should probably use the ```meta.broken``` attribute to indicate that.

But the expression for 4.8 has `darwin` in platforms and I didn't touch that part.

GrahamcOfBorg · 2018-10-21T18:48:07Z

Success on x86_64-linux (full log)

Attempted: llvm_5

Partial log (click to expand)

these paths will be fetched (37.43 MiB download, 216.54 MiB unpacked):
  /nix/store/8icl1pw8j94ybchwkb5lb89saklxg601-llvm-5.0.2
  /nix/store/cavwdv9nz27nj81q1v870dl8sbng3djq-llvm-5.0.2-lib
copying path '/nix/store/cavwdv9nz27nj81q1v870dl8sbng3djq-llvm-5.0.2-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/8icl1pw8j94ybchwkb5lb89saklxg601-llvm-5.0.2' from 'https://cache.nixos.org'...
/nix/store/8icl1pw8j94ybchwkb5lb89saklxg601-llvm-5.0.2

GrahamcOfBorg · 2018-10-21T18:50:13Z

Success on aarch64-linux (full log)

Attempted: llvm_5

Partial log (click to expand)

these paths will be fetched (33.27 MiB download, 200.78 MiB unpacked):
  /nix/store/xv06wq5jhlkj6vbmsmxfbkiv0drgypkg-llvm-5.0.2
  /nix/store/zwym0v67pjqxhvcgd3l2xrjy5sqxailg-llvm-5.0.2-lib
copying path '/nix/store/zwym0v67pjqxhvcgd3l2xrjy5sqxailg-llvm-5.0.2-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/xv06wq5jhlkj6vbmsmxfbkiv0drgypkg-llvm-5.0.2' from 'https://cache.nixos.org'...
/nix/store/xv06wq5jhlkj6vbmsmxfbkiv0drgypkg-llvm-5.0.2

oxij · 2018-10-29T09:53:08Z

ping? All of this builds fine on `x86_64-linux` and `meta` nitpicks can be fixed separately, IMHO.

7c6f434c

If there is an explicit fresh review request towards @xeji I want to allow some time for it to happen (or not). Ping me in 24 hours, I will merge provided @FRidh doesn't merge it by then (assuming @xeji doesn't have hash-affecting objections, of course)

xeji · 2018-10-29T19:08:53Z

I don't feel competent to review this since I'm not familiar with the details of our gcc and llvm expressions.

7c6f434c · 2018-10-29T19:17:23Z

I am not keeping a close track of the minor details in the expressions, but I can understand what these specific changes do, and I agree these are a mix of useful cleanups and neutral layout changes.

Ericson2314 · 2018-10-29T19:22:32Z

pkgs/development/compilers/llvm/5/llvm.nix

-  buildInputs = [ libxml2 libffi ]
-    # TODO(@Ericson2314): Remove next mass rebuild
-    ++ stdenv.lib.optionals (stdenv.isDarwin && stdenv.hostPlatform == stdenv.buildPlatform) [ libcxxabi ];
+  buildInputs = [ libxml2 libffi ];


Oh for the record this is because compiler-rt is now built separately and libcxxabi was just needed for that. Overall on the LLVM side, tools and runtime libraries are never built in the same derivation, which is good.

gcc-*: homogenize and cleanup expressions without a rebuild

e0b5919

Looks scary but it is a noop.

oxij requested a review from Ericson2314 as a code owner October 19, 2018 18:53

GrahamcOfBorg added 6.topic: stdenv Standard environment 10.rebuild-darwin-stdenv 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-linux: 11-100 labels Oct 19, 2018

oxij force-pushed the tree/mass-rebuild-complicated-cleanups branch from 84b48cb to 91ea34a Compare October 19, 2018 19:45

GrahamcOfBorg removed the 6.topic: stdenv Standard environment label Oct 19, 2018

veprbl reviewed Oct 19, 2018

View reviewed changes

oxij added 2 commits October 21, 2018 18:26

gcc-*: homogenize and cleanup expressions with a mass rebuild

eae890d

llvm_5: cleanup with a mass rebuild, but no idea why, and can not test

110f16e

oxij force-pushed the tree/mass-rebuild-complicated-cleanups branch from 91ea34a to 110f16e Compare October 21, 2018 18:31

FRidh requested review from xeji and 7c6f434c October 29, 2018 17:31

7c6f434c approved these changes Oct 29, 2018

View reviewed changes

xeji removed their request for review October 29, 2018 19:09

7c6f434c merged commit 4a85434 into NixOS:staging Oct 29, 2018

Ericson2314 reviewed Oct 29, 2018

View reviewed changes

oxij deleted the tree/mass-rebuild-complicated-cleanups branch August 12, 2023 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

treewide: mass rebuild complicated cleanups #48721

treewide: mass rebuild complicated cleanups #48721

oxij commented Oct 19, 2018

GrahamcOfBorg commented Oct 19, 2018

GrahamcOfBorg commented Oct 19, 2018

Ericson2314 commented Oct 19, 2018

oxij commented Oct 19, 2018 via email

GrahamcOfBorg commented Oct 19, 2018

GrahamcOfBorg commented Oct 19, 2018

veprbl Oct 19, 2018

veprbl Oct 19, 2018

FRidh commented Oct 21, 2018

oxij commented Oct 21, 2018 via email

GrahamcOfBorg commented Oct 21, 2018

GrahamcOfBorg commented Oct 21, 2018

oxij commented Oct 29, 2018 via email

7c6f434c left a comment

xeji commented Oct 29, 2018

7c6f434c commented Oct 29, 2018 •

edited

Ericson2314 Oct 29, 2018

treewide: mass rebuild complicated cleanups #48721

treewide: mass rebuild complicated cleanups #48721

Conversation

oxij commented Oct 19, 2018

Motivation for this change

Things done

GrahamcOfBorg commented Oct 19, 2018

GrahamcOfBorg commented Oct 19, 2018

Ericson2314 commented Oct 19, 2018

oxij commented Oct 19, 2018 via email

GrahamcOfBorg commented Oct 19, 2018

GrahamcOfBorg commented Oct 19, 2018

veprbl Oct 19, 2018

Choose a reason for hiding this comment

veprbl Oct 19, 2018

Choose a reason for hiding this comment

FRidh commented Oct 21, 2018

oxij commented Oct 21, 2018 via email

GrahamcOfBorg commented Oct 21, 2018

GrahamcOfBorg commented Oct 21, 2018

oxij commented Oct 29, 2018 via email

7c6f434c left a comment

Choose a reason for hiding this comment

xeji commented Oct 29, 2018

7c6f434c commented Oct 29, 2018 • edited

Ericson2314 Oct 29, 2018

Choose a reason for hiding this comment

7c6f434c commented Oct 29, 2018 •

edited