Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

patch-shebangs: respect cross compilation for 18.09 #46108

Merged
merged 3 commits into from Sep 11, 2018

Conversation

Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Sep 5, 2018

Motivation for this change

This is a version of #43833 made more conservative to only affect cross builds (well, anything enabling strictDeps). I hope that makes it conservative enough to land in 18.09. Sorry I didn't make this earlier, pre-fork.

The idea is that if strictDeps isn't defined, HOST_PATH is defined exactly the same as PATH. This means that patchShebangs, patchShebangs --build, and patchShebangs --host are all defined the same as each other and as patchShebangs is today.

I still like @matthewbauer's version for staging and 19.03 and beyond. The breakage it creates points us to things worth fixing, just not as part of the 18.09 deadline!

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/5nlybjij7qvxaf9wj8zmd06j4f1fksj6-help2man-1.47.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/z8ihiai14dik7af92k18a4ymygmvxksa-binutils-wrapper-2.30.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/hbcf7dzg9ajhkwsvx9cxjp143w02814q-diffutils-3.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/1zkyfc7cmkd6lg475777m3kjbm6y19ra-findutils-4.6.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/11cc22sh46nv3qfyv6b0mbf31chz724d-libtool-2.4.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/0ax7048h5k873ghkgj22nzdarw215xh4-gcc-wrapper-7.3.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/vxhj0jwm82dbw43bpykalzbmifgv1xpy-hook.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/hnjimxn482yismhqqic3psvvp5f4in86-patch-2.7.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/kvz4sdp8s9xd4bcy1wkv4xqx3lqfnkas-stdenv-linux.drv': 23 dependencies couldn't be built
error: build of '/nix/store/kvz4sdp8s9xd4bcy1wkv4xqx3lqfnkas-stdenv-linux.drv' failed

@matthewbauer
Copy link
Member

matthewbauer commented Sep 5, 2018

Maybe we should merge #43833 first just to make sure the changes work in all cases. I don't want to break patch shebangs in 18.09 (even if it looks harmless). This can be backported a little bit later.

@Ericson2314
Copy link
Member Author

@matthewbauer err you mean merge this into staging first? Your can indeed break native builds, but this is only capable of breaking cross builds which somehow relied on the broken behavior.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

shrinking /nix/store/47wvfxxmipwzsjhbr7qp39vjm0wai0q6-findutils-4.6.0/bin/find
gzipping man pages under /nix/store/47wvfxxmipwzsjhbr7qp39vjm0wai0q6-findutils-4.6.0/share/man/
strip is /nix/store/8zbl35kgg8v7gircl3pvy9027vslll1q-bootstrap-tools/bin/strip
stripping (with command strip and flags -S) in /nix/store/47wvfxxmipwzsjhbr7qp39vjm0wai0q6-findutils-4.6.0/libexec  /nix/store/47wvfxxmipwzsjhbr7qp39vjm0wai0q6-findutils-4.6.0/bin
checking for references to /build in /nix/store/47wvfxxmipwzsjhbr7qp39vjm0wai0q6-findutils-4.6.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/mfg6a2bmwjyj79w1f3s4813szixi337f-findutils-4.6.0-info
strip is /nix/store/8zbl35kgg8v7gircl3pvy9027vslll1q-bootstrap-tools/bin/strip
checking for references to /build in /nix/store/mfg6a2bmwjyj79w1f3s4813szixi337f-findutils-4.6.0-info...
building '/nix/store/kwgrkg6z99nq9j2flashhc9jl0gbk207-stdenv-linux.drv'...
/nix/store/syn3p6a0xk1livqsqv41avdddh7660ww-stdenv-linux

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-linux (full log)

Attempted: dhcpcd, stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/5kb2175p39bi3j41cl16kxv0nnw62izz-meson-0.46.1.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/kzxsm8b0908952w9g7n4wg25h203qvk8-python3.6-lxml-4.2.4.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/f6zwwvpvinm4zq512g987gs3wc7l5b0d-gnome-doc-utils-0.20.10.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/fh8jsrambn0aq06rvigyz87vb6kghsqg-python3-3.6.6-env.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/g8mi9wp6sn9lnczqb6qx3sqbjgm625z5-shadow-4.6.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/lpx53k26nfx9h1jc9bcyh3hjzv1j0l9k-util-linux-2.32.1.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ypx34axxbd3xqv9j936l3flb0p8zy716-glib-2.56.0.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/ajn3f20hd6nmkf6vbbxhx475bwwm14sk-systemd-239.drv': 42 dependencies couldn't be built
cannot build derivation '/nix/store/qkj11c06is3p2a1w0kdny4clk7q5aqk5-dhcpcd-7.0.8.drv': 4 dependencies couldn't be built
error: build of '/nix/store/kvz4sdp8s9xd4bcy1wkv4xqx3lqfnkas-stdenv-linux.drv', '/nix/store/qkj11c06is3p2a1w0kdny4clk7q5aqk5-dhcpcd-7.0.8.drv' failed

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: dhcpcd, stdenv

Partial log (click to expand)

shrinking /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/sbin/dhcpcd
gzipping man pages under /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/share/man/
strip is /nix/store/830hkkm6q88wx7zfrilbkl9xmv5n8yrg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/lib  /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/libexec  /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/sbin
patching script interpreter paths in /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8
/nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/libexec/dhcpcd-run-hooks: interpreter directive changed from "/bin/sh" to "/nix/store/1sdylxi5pq4ja5dr5g4m83bv1m3hc92f-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8...
moving /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/sbin/* to /nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8/bin
/nix/store/a0pk7218d5skq55yg8a4sp820l22z9zx-dhcpcd-7.0.8
/nix/store/syn3p6a0xk1livqsqv41avdddh7660ww-stdenv-linux

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: stdenv

Partial log (click to expand)

[ 56%] Building CXX object lib/Target/Hexagon/CMakeFiles/LLVMHexagonCodeGen.dir/HexagonCFGOptimizer.cpp.o
[ 57%] Building CXX object lib/Target/Hexagon/CMakeFiles/LLVMHexagonCodeGen.dir/HexagonCommonGEP.cpp.o
[ 57%] Building CXX object lib/Target/Hexagon/CMakeFiles/LLVMHexagonCodeGen.dir/HexagonConstPropagation.cpp.o
[ 57%] Building CXX object lib/Target/Hexagon/CMakeFiles/LLVMHexagonCodeGen.dir/HexagonCopyToCombine.cpp.o
building of '/nix/store/dzbnwpxv7xwhr2yicsd44ax9mxmpa5rd-llvm-5.0.2.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/838bnnmcj14kibbfr7z3w0yvyik5nd8k-clang-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/3f0agwbd4842fvhck86hl4c4kydp2zxz-compiler-rt-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/724bry6jw2a8vb0abq6gxr4l82jg0cad-clang-wrapper-5.0.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/z9xdq92n2p9hbzkq1zqdj0fw5gni020d-stdenv-darwin.drv': 4 dependencies couldn't be built
error: build of '/nix/store/z9xdq92n2p9hbzkq1zqdj0fw5gni020d-stdenv-darwin.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: stdenv

The following builds were skipped because they don't evaluate on x86_64-darwin: dhcpcd

Partial log (click to expand)

[ 66%] Building NVPTXGenSubtargetInfo.inc...
[ 66%] Updating NVPTXGenSubtargetInfo.inc...
[ 66%] Built target NVPTXCommonTableGen
Scanning dependencies of target LLVMNVPTXCodeGen
building of '/nix/store/dzbnwpxv7xwhr2yicsd44ax9mxmpa5rd-llvm-5.0.2.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/838bnnmcj14kibbfr7z3w0yvyik5nd8k-clang-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/3f0agwbd4842fvhck86hl4c4kydp2zxz-compiler-rt-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/724bry6jw2a8vb0abq6gxr4l82jg0cad-clang-wrapper-5.0.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/z9xdq92n2p9hbzkq1zqdj0fw5gni020d-stdenv-darwin.drv': 4 dependencies couldn't be built
error: build of '/nix/store/z9xdq92n2p9hbzkq1zqdj0fw5gni020d-stdenv-darwin.drv' failed

@Ericson2314
Copy link
Member Author

OK I think I see. It's fine if staging breaks a little since we want to fix the misbehaving packages anyways, and we still get some sanity checking of does the code do the right thing. 👍.

Copy link
Member

@matthewbauer matthewbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to go now.

@Ericson2314
Copy link
Member Author

Alright!

Ericson2314 and others added 3 commits September 11, 2018 16:15
This ensures this change is safe last minute for 18.09. Master won't
have this.
This hopefully makes patchShebangs respect cross compilation. It
introduces the concept of the HOST_PATH. Nothing is ever executed on
it but instead used as a way to get the proper path using ‘command
-v’. Needs more testing.

/cc @Ericson2314 @dtzWill

Fixes NixOS#33956
Fixes NixOS#21138

(Modified backport of f069423. See
previous commit to understand the differences between this and the
original.)
@Ericson2314
Copy link
Member Author

Changed the order so that #46542 for regular staging can share a commit.

@Ericson2314 Ericson2314 merged commit 6877b4c into NixOS:staging-18.09 Sep 11, 2018
@Ericson2314 Ericson2314 deleted the cross-patch-shebangs branch September 11, 2018 20:41
@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-linux (full log)

Attempted: dhcpcd, stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/5kb2175p39bi3j41cl16kxv0nnw62izz-meson-0.46.1.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/kzxsm8b0908952w9g7n4wg25h203qvk8-python3.6-lxml-4.2.4.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/f6zwwvpvinm4zq512g987gs3wc7l5b0d-gnome-doc-utils-0.20.10.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/fh8jsrambn0aq06rvigyz87vb6kghsqg-python3-3.6.6-env.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/g8mi9wp6sn9lnczqb6qx3sqbjgm625z5-shadow-4.6.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/lpx53k26nfx9h1jc9bcyh3hjzv1j0l9k-util-linux-2.32.1.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ypx34axxbd3xqv9j936l3flb0p8zy716-glib-2.56.0.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/ajn3f20hd6nmkf6vbbxhx475bwwm14sk-systemd-239.drv': 42 dependencies couldn't be built
cannot build derivation '/nix/store/qkj11c06is3p2a1w0kdny4clk7q5aqk5-dhcpcd-7.0.8.drv': 4 dependencies couldn't be built
error: build of '/nix/store/kvz4sdp8s9xd4bcy1wkv4xqx3lqfnkas-stdenv-linux.drv', '/nix/store/qkj11c06is3p2a1w0kdny4clk7q5aqk5-dhcpcd-7.0.8.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: stdenv

The following builds were skipped because they don't evaluate on x86_64-darwin: dhcpcd

Partial log (click to expand)

cannot build derivation '/nix/store/34w061dgh6rb31lhs60mg93v82qh1arb-cctools-port-895.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/knh9nayqr6y15r2mq3gzqvlhakh3kk28-hook.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/ygc58kc2xzf3vaxn98ygzd4rm8qp7fci-ICU-osx-10.10.5.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/806gwymraxmd8favdz3qs13c0n79fvsh-cctools-binutils-darwin.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/cfk18hqiw8y0h9fw6inj7ivdzbxg7dk7-gnutar-1.30.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/dylfb5nmsmczaih39crprqg91nzlgjk8-CF-osx-10.10.5.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/k7mrps05qnxvlghh2xg3ji0pdf3ssylc-cctools-binutils-darwin-wrapper.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/724bry6jw2a8vb0abq6gxr4l82jg0cad-clang-wrapper-5.0.2.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/z9xdq92n2p9hbzkq1zqdj0fw5gni020d-stdenv-darwin.drv': 36 dependencies couldn't be built
error: build of '/nix/store/z9xdq92n2p9hbzkq1zqdj0fw5gni020d-stdenv-darwin.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on aarch64-linux (full log)

Attempted: dhcpcd, stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/fh5aqvgjnm7mkg8v8z1yg4a91z45whx5-libmicrohttpd-0.9.59.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/zi5j5xfi581j0rlqy749l2fz08cy3a4l-pciutils-3.6.2.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/y2rj7qn0msfrdk8x44pa9lwiqi9prbzn-python3-3.6.6-env.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/h9rf1wrlnplq213f5wp4kximj60ani0z-gnu-efi-3.0.8.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/8xp26zjfdmr2gzznplz98acgn7qmifmg-shadow-4.6.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/aj17hdy4p1z3bwgvjhj0gr6yfcfhydas-util-linux-2.32.1.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/7yl24ql79vvzh8m32ql9vb4j26xl3kzk-glib-2.56.0.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/jxakfrlwmazsjdipy6338shqzsql93f5-systemd-239.drv': 42 dependencies couldn't be built
cannot build derivation '/nix/store/m85rbxsyzfykjjn32byficigzw1mxbfy-dhcpcd-7.0.8.drv': 4 dependencies couldn't be built
error: build of '/nix/store/kwgrkg6z99nq9j2flashhc9jl0gbk207-stdenv-linux.drv', '/nix/store/m85rbxsyzfykjjn32byficigzw1mxbfy-dhcpcd-7.0.8.drv' failed

@matthewbauer
Copy link
Member

@Ericson2314 Looking at this now, I think I have found a bug in this. Lots of packages have bash scripts in their outputs but don't directly list bash as an input. Instead they get it from initialPath. But initialPath goes into PATH but not HOST_PATH. We need to find a way to make sure that a runtime Bash gets in HOST_PATH. I think defaultBuildInputs is the easiest way to go.

But for now, this will break in unexpected ways. I am going to revert this in staging-18.09 just to be safe.

matthewbauer added a commit that referenced this pull request Sep 16, 2018
…ebangs"

This reverts commit 6877b4c, reversing
changes made to 69b6db0.
@Ericson2314
Copy link
Member Author

Ah man, all those little corner cases. Sure complicates everything. Good catch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants