Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix breaking changes to make-derivation #51183

Merged

Conversation

matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Nov 28, 2018

Motivation for this change

A few breaking changes have been introduced to make-derivation recently. IMO make-derivation shouldn't ever change in what it accepts from release to release. There's too many places where this is used both internally and externally.

So this restores a few things:

  • Automatic adding host suffixes to name when a cross system exists. This should not happen with simple things like substituteAll where name is often used for things like binary names. Try cross compiling nixos-install for example.
  • Don't assert when version is set but pname isn't. Again lots of places use substituteAll and the version attr. No reason to assert here.
  • Don't assert when separateDebugInfo is on and we aren't on Linux. While the separate debug info only works on Linux, that doesn't mean that non-Linux stuff is "broken". You just can't generate debug outputs in this way.

version is set in lots of places but might not need to be in a name.

Alternative to NixOS#50364.
When separateDebugInfo = true & !stdenv.hostPlatform.isLinux, we
always gave an error. There is no reason to do this. Instead, just
don’t add separate debug info when we aren’t on Linux.
Some trivial builders use the name attr to choose the exec name
produced. For example nixos-install,

  nixos-install = makeProg {
    name = "nixos-install";
    src = ./nixos-install.sh;
    nix = config.nix.package.out;
    path = makeBinPath [ nixos-enter ];
  };

When cross compiling, this puts the prog in,

  /bin/nixos-install-powerpc64le-unknown-linux-gnu

fixedOutputDrv = attrs ? outputHash;
noNonNativeDeps = builtins.length (depsBuildTarget ++ depsBuildTargetPropagated
++ depsHostHost ++ depsHostHostPropagated
++ buildInputs ++ propagatedBuildInputs
++ depsTargetTarget ++ depsTargetTargetPropagated) == 0;
runtimeSensativeIfFixedOutput = fixedOutputDrv -> !noNonNativeDeps;
dontAddHostSuffix = attrs ? outputHash && !noNonNativeDeps || stdenv.cc == null;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ericson2314 Does this preserve the original condition? I'm confused as to whether you want the suffix when you have non-native deps or not?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sorry my original booleans are a bit obtuse because I also wanted to use this to make a warning saying fixed-output derivations shouldn't have run-time deps.

In the case that they do, the suffix should appear because the run-time deps make the derivation host-sensative.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sensative

-> "sensitive" !!


Typo is no big deal, but thought I'd say something since it's the spelling also used for the variable name 😿 .

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that too! I also like dontAddHostSuffix because it's a little bit clearer on what is actually happening.

@hedning hedning mentioned this pull request Nov 29, 2018
9 tasks
@matthewbauer
Copy link
Member Author

@GrahamcOfBorg build stdenv hello pkgsCross.aarch64-multiplatform.hello

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: stdenv, hello, pkgsCross.aarch64-multiplatform.hello

Partial log (click to expand)

make[2]: Leaving directory '/build/hello-2.10'
make[1]: Leaving directory '/build/hello-2.10'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/c4jvfirsaswrx00231hky59ndzr6idcm-hello-2.10
shrinking /nix/store/c4jvfirsaswrx00231hky59ndzr6idcm-hello-2.10/bin/hello
gzipping man pages under /nix/store/c4jvfirsaswrx00231hky59ndzr6idcm-hello-2.10/share/man/
strip is /nix/store/0z21vv6i3bg0p5k7pf0cg5qzv2psnpr7-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/c4jvfirsaswrx00231hky59ndzr6idcm-hello-2.10/bin
patching script interpreter paths in /nix/store/c4jvfirsaswrx00231hky59ndzr6idcm-hello-2.10
checking for references to /build/ in /nix/store/c4jvfirsaswrx00231hky59ndzr6idcm-hello-2.10...

@matthewbauer matthewbauer merged commit e683417 into NixOS:staging Nov 30, 2018
@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-linux (full log)

Attempted: stdenv, hello, pkgsCross.aarch64-multiplatform.hello

Partial log (click to expand)

cannot build derivation '/nix/store/lw8bzgpc3l6cl1gpb4gxwibwc34kk8cf-aarch64-unknown-linux-gnu-stage-static-gcc-debug-7.3.0.drv': 13 dependencies couldn't be built
cannot build derivation '/nix/store/hpzrkr428mbgq66qflv3q545l5jp2nfn-aarch64-unknown-linux-gnu-stage-static-gcc-debug-wrapper-7.3.0.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/pnlr81awm2wn51r7wz6fr6ggppww3gxk-stdenv-linux.drv': 17 dependencies couldn't be built
cannot build derivation '/nix/store/18s6hyk5ss8x7jnq1nia7qvdn5sywc8b-glibc-2.27-aarch64-unknown-linux-gnu.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/c40x2pm68m3q16ia21i62fzqnj5lrwxc-aarch64-unknown-linux-gnu-binutils-wrapper-2.30.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/6j05c47xqg2dd1wshqqqwqmpz2a9kf44-aarch64-unknown-linux-gnu-stage-final-gcc-debug-7.3.0.drv': 14 dependencies couldn't be built
cannot build derivation '/nix/store/7mbkxa3dgsqjz4x6nmk1wj05r1zib8kl-aarch64-unknown-linux-gnu-stage-final-gcc-debug-wrapper-7.3.0.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/6k9rxgwyva1c5j9w56dw0c3fl1lly3pi-stdenv-linux.drv': 17 dependencies couldn't be built
cannot build derivation '/nix/store/cj07wpi5zckdhyzx89fafqb51zv7hjad-hello-2.10-aarch64-unknown-linux-gnu.drv': 2 dependencies couldn't be built
error: build of '/nix/store/75cma6l5wsppbqfxzg7c6rxw3ygah28b-hello-2.10.drv', '/nix/store/cj07wpi5zckdhyzx89fafqb51zv7hjad-hello-2.10-aarch64-unknown-linux-gnu.drv', '/nix/store/g4g3mxly9v75xnici5xid1i45044xj33-stdenv-linux.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: stdenv, hello, pkgsCross.aarch64-multiplatform.hello

Partial log (click to expand)

cannot build derivation '/nix/store/743m2cl1hb56npvbz35pi3zpf4601y22-aarch64-unknown-linux-gnu-stage-static-gcc-debug-7.3.0.drv': 15 dependencies couldn't be built
cannot build derivation '/nix/store/3ydm1bw7vpgid1adf6syxs8ii29pzlqh-aarch64-unknown-linux-gnu-stage-static-gcc-debug-wrapper-7.3.0.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/7ync4k4j2msdbzmvycsar1nqbgczk1fh-stdenv-darwin.drv': 19 dependencies couldn't be built
cannot build derivation '/nix/store/095xkpjm2mqcilfm3k06879l6hiwnl1a-glibc-2.27-aarch64-unknown-linux-gnu.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/6w5zhibcyg3k9mnki336qdbaq9hqc6wc-aarch64-unknown-linux-gnu-binutils-wrapper-2.30.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/g1nh8gbcx2qimrmpz60id4lxgpza6ms2-aarch64-unknown-linux-gnu-stage-final-gcc-debug-7.3.0.drv': 16 dependencies couldn't be built
cannot build derivation '/nix/store/4rfi8s0qsxxkrxpvk737nyn12pbflw2r-aarch64-unknown-linux-gnu-stage-final-gcc-debug-wrapper-7.3.0.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/34y3w4zp8yi8pxncsf936sdrrk9hp6ka-stdenv-darwin.drv': 19 dependencies couldn't be built
cannot build derivation '/nix/store/v42aygdsiray7yl2kj3lbjld0bk4f7jw-hello-2.10-aarch64-unknown-linux-gnu.drv': 2 dependencies couldn't be built
error: build of '/nix/store/134bg2x6pwf3yx7g2l0s8mkc1ghwqcsb-hello-2.10.drv', '/nix/store/17jilrg4arc0xhgni6012bypgi1a0bpy-stdenv-darwin.drv', '/nix/store/v42aygdsiray7yl2kj3lbjld0bk4f7jw-hello-2.10-aarch64-unknown-linux-gnu.drv' failed

@Ekleog
Copy link
Member

Ekleog commented Nov 30, 2018

cc @Synthetica9 @dtzWill (and @Ericson2314 has already been pinged) whose changes have been reverted here, if I read correctly the blame.

Meta-point (I'm not familiar with the changes this PR makes, so this is not an opinion on these specific changes): I personally totally disagree with

IMO make-derivation shouldn't ever change in what it accepts from release to release. There's too many places where this is used both internally and externally.

My reasons are simple:

  • Internal uses of make-derivation will be found by hydra and ZHF before the next release
  • External uses of make-derivation that use unstable should expect breakage (it's “unstable”, after all), and we've explicitly allowed ourselves to break backwards-compatibility between releases
  • If we don't allow ourselves to change it, then code sclerosis kicks in and there is no way to repay the technical debt we're accumulating in Nixpkgs.

Anyway, all that to say that a decision of “make-derivation should be stable even between releases” vs. “make-derivation can be backwards-compatibility-broken between releases” should be done via a RFC, once the RFC-process-update RFC will be done, so that all arguments in favor and against are exposed and the decision can be considered as chosen by the community as a whole.

@Ericson2314
Copy link
Member

Yes I had no problem with the specific changes here, but also disagree with that justification as @Ekleog says.

@infinisil
Copy link
Member

Commit a56fe05 is causing (bisected by @cleverca22, after wanting to demonstrate cross comp in #nixos):

$ nix-instantiate -A pkgsCross.mingwW64.hello --show-trace
error: while evaluating 'optionalString' at /home/infinisil/src/nixpkgs/lib/strings.nix:180:5, called from /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:58:
while evaluating the attribute 'cc' at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/default.nix:145:14:
while evaluating the attribute 'gcc' at /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:6728:3:
while evaluating 'lowPrio' at /home/infinisil/src/nixpkgs/lib/meta.nix:52:13, called from /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:6869:10:
while evaluating 'addMetaAttrs' at /home/infinisil/src/nixpkgs/lib/meta.nix:15:28, called from /home/infinisil/src/nixpkgs/lib/meta.nix:52:18:
while evaluating 'wrapCC' at /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:7612:12, called from /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:6869:19:
while evaluating 'wrapCCWith' at /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:7591:5, called from /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:7612:16:
while evaluating 'callPackageWith' at /home/infinisil/src/nixpkgs/lib/customisation.nix:108:35, called from /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:7600:7:
while evaluating 'makeOverridable' at /home/infinisil/src/nixpkgs/lib/customisation.nix:67:24, called from /home/infinisil/src/nixpkgs/lib/customisation.nix:112:8:
while evaluating anonymous function at /home/infinisil/src/nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1, called from /home/infinisil/src/nixpkgs/lib/customisation.nix:69:12:
while evaluating the derivation attribute 'name' at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
while evaluating 'optionalString' at /home/infinisil/src/nixpkgs/lib/strings.nix:180:5, called from /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:58:
while evaluating the attribute 'cc' at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/default.nix:145:14:
while evaluating the attribute 'gccCrossStageStatic' at /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:6793:3:
while evaluating 'wrapCCWith' at /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:7591:5, called from /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:6802:8:
while evaluating 'callPackageWith' at /home/infinisil/src/nixpkgs/lib/customisation.nix:108:35, called from /home/infinisil/src/nixpkgs/pkgs/top-level/all-packages.nix:7600:7:
while evaluating 'makeOverridable' at /home/infinisil/src/nixpkgs/lib/customisation.nix:67:24, called from /home/infinisil/src/nixpkgs/lib/customisation.nix:112:8:
while evaluating anonymous function at /home/infinisil/src/nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1, called from /home/infinisil/src/nixpkgs/lib/customisation.nix:69:12:
while evaluating the derivation attribute 'name' at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
while evaluating the attribute 'name' at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
while evaluating 'optionalString' at /home/infinisil/src/nixpkgs/lib/strings.nix:180:5, called from /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:58:
while evaluating the attribute 'cc' at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/default.nix:145:14:
infinite recursion encountered, at /home/infinisil/src/nixpkgs/pkgs/stdenv/generic/default.nix:145:14

@matthewbauer
Copy link
Member Author

Yes tracking that in #53587.

@matthewbauer matthewbauer deleted the make-derivation-no-assertions branch February 22, 2019 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants