Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llvm-packages: get rid of extra build depedencies for manpages #26818

Merged
merged 3 commits into from Jun 26, 2017

Conversation

LnL7
Copy link
Member

@LnL7 LnL7 commented Jun 24, 2017

Motivation for this change

Fixes #26756

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

Estimating rebuild amount by counting changed Hydra jobs.
  31 x86_64-darwin
3658 x86_64-linux

@mention-bot
Copy link

@LnL7, thanks for your PR! By analyzing the history of the files in this pull request, we identified @dtzWill, @copumpkin and @acowley to be potential reviewers.

@@ -1,4 +1,4 @@
{ newScope, stdenv, cmake, libxml2, python2, isl, fetchurl, overrideCC, wrapCC, darwin, ccWrapperFun }:
{ lowPrio, newScope, stdenv, cmake, libxml2, python2, isl, fetchurl, overrideCC, wrapCC, darwin, ccWrapperFun }:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be missing something but there's no reference to lowPrio

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, forgot to apply it to the *-manpages attributes

@copumpkin
Copy link
Member

Looks good, although it's a pity we have to do a whole new 1h+ build to get some documentation. Someone who understands the LLVM build internals a bit better can probably optimize that a bit though.

@copumpkin
Copy link
Member

Should this go against staging? Or shall we just take the mass-darwin-rebuild?

@LnL7 LnL7 changed the base branch from master to staging June 26, 2017 18:26
@LnL7 LnL7 merged commit d76bc8e into NixOS:staging Jun 26, 2017
@LnL7 LnL7 deleted the llvm-manpages branch June 26, 2017 19:40
@copumpkin
Copy link
Member

Thanks @LnL7!

@vcunat
Copy link
Member

vcunat commented Jul 3, 2017

@LnL7: this makes all darwin stuff broken, so Hydra hasn't been building anything for it on staging. Can you/someone fix this fast? (I don't see directly what packages cause it.) Otherwise I suggest to revert for now and leave it for another staging iteration.

@vcunat
Copy link
Member

vcunat commented Jul 3, 2017

@LnL7
Copy link
Member Author

LnL7 commented Jul 3, 2017

Are you sure it was caused by these changes? I merged this in a week ago and tested it with my local hydra setup.

@vcunat
Copy link
Member

vcunat commented Jul 3, 2017

Oh, I'm sorry, I messed this up. The real culprit is 4ac1901. (Fortunately, I can evaluate easily without having a Darwin machine.)

@edolstra
Copy link
Member

This breaks the NixOS channel update:

Jul 13 08:31:26 webserver update-nixos-unstable-start[7036]: error: path ‘/nix/store/042yklr7p47rqcq56yr5s6qh0xslhlbp-llvm-4.0.1-man’ is not valid
Jul 13 08:31:26 webserver update-nixos-unstable-start[7036]: Died at /nix/store/rliwwbr2nnszan9nxp76lmx7js3kriq8-nixos-channel-scripts/bin/.mirror-nixos-branch-wrapped line 159.

The reason is that this line:

drv // { man = drv-manpages.man; outputs = drv.outputs ++ ["man"]; };

makes Hydra think that the llvm derivation has a man output, but it doesn't. See http://hydra.nixos.org/build/56151666#tabs-details, which has /nix/store/042yklr7p47rqcq56yr5s6qh0xslhlbp-llvm-4.0.1-man as an output. However, that path never gets built because it's not actually an output of the .drv.

A possible fix is to have hydra-eval-jobs looks at the .drv instead of the outputs of the evaluated derivation attrset.

@vcunat
Copy link
Member

vcunat commented Jul 13, 2017

Hmm, maybe this approach to override .outputs is too hacky. There might be more places that assume all outputs come from the same derivation, perhaps even in Hydra. (Just the web UI just shows the derivation and output paths.)

We certainly want to unblock the channel fast – any immediate ideas? If not, I'd probably mark this llvmPackages set without recurseForDerivations and add a parallel set where llvm isn't hacked this way.

@edolstra
Copy link
Member

Easiest fix is to remove ++ ["man"]. Then you can still get the manpages via llvm.man.

@vcunat
Copy link
Member

vcunat commented Jul 13, 2017

Oh, right. I thought of meta.outputsToInstall and such, but I don't see it checked against .outputs on any relevant places (only attribute existence is checked), and nix-env checks the real outputs of the derivation (like Hydra) so there's no help anyway.

vcunat added a commit that referenced this pull request Jul 13, 2017
@vcunat
Copy link
Member

vcunat commented Jul 13, 2017

Pushed as 1e1472e.

@copumpkin
Copy link
Member

The bigger concern for me is if nix-env doesn't pay attention to it as @vcunat says, then it's not a great user experience.

@vcunat
Copy link
Member

vcunat commented Jul 13, 2017

The main UX blocker ATM for multiple outputs (e.g. man) is #24717, I think.

@vcunat
Copy link
Member

vcunat commented Jul 15, 2017

@edolstra: there's apparently some other blocker I can't see, as that very commit has successfully passed the tested job and there's still no channel bump.

@vcunat
Copy link
Member

vcunat commented Jul 15, 2017

Maybe it's related to Hydra's evaluator complaining about being out of space?

@edolstra
Copy link
Member

Both nixos.org and Hydra has a disk full. Both seem fine now. However, the nixos-unstable channel cannot update:

Jul 17 13:15:12 webserver update-nixos-unstable-start[28946]: release is ‘nixos-17.09pre110737.02a268430e’ (build 56231260), eval is 1375007, prefix is nixos/unstable/nixos-17.09pre110737.02a268430e, Git commit is 02a268430e13061aad441ec4a28579d46af79e33
Jul 17 13:15:12 webserver update-nixos-unstable-start[28946]: Fetching origin
Jul 17 13:17:28 webserver update-nixos-unstable-start[28946]: From https://github.com/NixOS/nixpkgs
Jul 17 13:17:28 webserver update-nixos-unstable-start[28946]:  * [new branch]            nginx-gzip-vary -> origin/nginx-gzip-vary
Jul 17 13:17:28 webserver update-nixos-unstable-start[28946]: Auto packing the repository in background for optimum performance.
Jul 17 13:17:28 webserver update-nixos-unstable-start[28946]: See "git help gc" for manual housekeeping.
Jul 17 13:17:28 webserver update-nixos-unstable-start[28946]: Auto packing the repository in background for optimum performance.
Jul 17 13:17:28 webserver update-nixos-unstable-start[28946]: See "git help gc" for manual housekeeping.
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: To https://github.com/NixOS/nixpkgs-channels.git
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]:  ! [rejected]              02a268430e13061aad441ec4a28579d46af79e33 -> nixos-unstable (non-fast-forward)
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: error: failed to push some refs to 'https://github.com/NixOS/nixpkgs-channels.git'
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: hint: Updates were rejected because a pushed branch tip is behind its remote
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: hint: counterpart. Check out this branch and integrate the remote changes
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: hint: (e.g. 'git pull ...') before pushing again.
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Jul 17 13:17:33 webserver update-nixos-unstable-start[28946]: Died at /nix/store/rliwwbr2nnszan9nxp76lmx7js3kriq8-nixos-channel-scripts/bin/.mirror-nixos-branch-wrapped line 88.

Looks like the channel would go back in time. Maybe some builds got restarted?

@domenkozar
Copy link
Member

Yes, I restarted the failing builds as an attempt to get this going forward. I don't understand why channel would go back in time, since it's 13 days old?

@vcunat
Copy link
Member

vcunat commented Jul 17, 2017

@edolstra: nixos-unstble branch in nixpkgs-channels is at a rather new commit 1e1472e that doesn't correspond to what's in http://nixos.org/channels/nixos-unstable. I suspect all this is due to non-atomicity of the mirroring script (in combination with some error).

@edolstra
Copy link
Member

edolstra commented Jul 17, 2017

From earlier in the journal:

Jul 14 06:15:40 webserver update-nixos-unstable-start[31959]: release is ‘nixos-17.09pre110815.1e1472ed93’ (build 56296734), eval is 1375288, prefix is nixos/unstable/nixos-17.09pre110815.1e1472ed93, Git commit is 1e1472ed93e1256dfdac4908ae8edcb90ceff21c
Jul 14 06:15:41 webserver update-nixos-unstable-start[31959]: Fetching origin
Jul 14 06:15:43 webserver update-nixos-unstable-start[31959]: Auto packing the repository in background for optimum performance.
Jul 14 06:15:43 webserver update-nixos-unstable-start[31959]: See "git help gc" for manual housekeeping.
Jul 14 06:15:43 webserver update-nixos-unstable-start[31959]: Auto packing the repository in background for optimum performance.
Jul 14 06:15:43 webserver update-nixos-unstable-start[31959]: See "git help gc" for manual housekeeping.
Jul 14 06:15:55 webserver update-nixos-unstable-start[31959]: To https://github.com/NixOS/nixpkgs-channels.git
Jul 14 06:15:55 webserver update-nixos-unstable-start[31959]:    02a268430e..1e1472ed93  1e1472ed93e1256dfdac4908ae8edcb90ceff21c -> nixos-unstable
Jul 14 06:16:02 webserver update-nixos-unstable-start[31959]: downloading /nix/store/vg4kiyb686gj8210ch0vwhz53a0h7nda-nixos-channel-17.09pre110815.1e1472ed93/tarballs/nixos-17.09pre110815.1e1472ed93.tar.xz to /data/releases/tmp/release-nixos-unstable/nixos-17.09pre110815.1e1472ed93/nixexprs.tar.xz...
Jul 14 06:16:07 webserver update-nixos-unstable-start[31959]: downloading /nix/store/gg6r3q688c9xf99dmdwhv6x80p6dylsw-nixos-minimal-17.09pre110815.1e1472ed93-x86_64-linux.iso/iso/nixos-minimal-17.09pre110815.1e1472ed93-x86_64-linux.iso to /data/releases/tmp/release-nixos-unstable/nixos-17.09pre110815.1e1472ed93/nixos-minimal-17.09pre110815.1e1472ed93-x86_64-linux.iso...
Jul 14 06:16:38 webserver update-nixos-unstable-start[31959]: downloading /nix/store/0caqaz6rg86908c6gxjc4cifmg2cc8q1-nixos-minimal-17.09pre110815.1e1472ed93-i686-linux.iso/iso/nixos-minimal-17.09pre110815.1e1472ed93-i686-linux.iso to /data/releases/tmp/release-nixos-unstable/nixos-17.09pre110815.1e1472ed93/nixos-minimal-17.09pre110815.1e1472ed93-i686-linux.iso...
Jul 14 06:17:13 webserver update-nixos-unstable-start[31959]: warning: unable to download ‘https://cache.nixos.org/nar/120ylni7fwsknrgabxsp7z9fsmxrcdjic05jm634qdankswck53z.nar.xz’: HTTP error 200 (curl error: Stream error in the HTTP/2 framing layer); retrying in 336 ms
Jul 14 06:17:38 webserver update-nixos-unstable-start[31959]: downloading /nix/store/c0vr5dc0564q7hx38lzsb44p0lqbixfy-nixos-graphical-17.09pre110815.1e1472ed93-x86_64-linux.iso/iso/nixos-graphical-17.09pre110815.1e1472ed93-x86_64-linux.iso to /data/releases/tmp/release-nixos-unstable/nixos-17.09pre110815.1e1472ed93/nixos-graphical-17.09pre110815.1e1472ed93-x86_64-linux.iso...
Jul 14 06:19:24 webserver update-nixos-unstable-start[31959]: downloading /nix/store/vdpk3b74pasjcxlk3qb3z0si985a9yl0-nixos-ova-17.09pre110815.1e1472ed93-x86_64-linux/nixos-17.09pre110815.1e1472ed93-x86_64-linux.ova to /data/releases/tmp/release-nixos-unstable/nixos-17.09pre110815.1e1472ed93/nixos-17.09pre110815.1e1472ed93-x86_64-linux.ova...
Jul 14 06:23:16 webserver update-nixos-unstable-start[31959]: error: path ‘/nix/store/042yklr7p47rqcq56yr5s6qh0xslhlbp-llvm-4.0.1-man’ is not valid
Jul 14 06:23:16 webserver update-nixos-unstable-start[31959]: Died at /nix/store/3hdvsww1aqbq0d6dyqa1agfx2q5xc42c-nixos-channel-scripts/bin/.mirror-nixos-branch-wrapped line 159.

So it updated the nixpkgs-channels to eval 1375288, but the rest of the release failed. Then the restart made 1375288 no longer the latest release on Hydra so it went back in time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants