nixos,nixpkgs: only build essentials on i686 #27923
Conversation
|
@grahamc, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @Ericson2314 and @domenkozar to be potential reviewers. |
|
Looks correct, will merge when either @fpletz or @domenkozar have sanity-checked this. |
|
You can use |
| (all nixpkgs.emacs) | ||
| (all nixpkgs.jdk) | ||
| (all allSupportedNixpkgs.emacs) | ||
| (all allSupportedNixpkgs.jdk) |
There was a problem hiding this comment.
This looks like the wrong place to build these packages. Shouldn't we rather move this to nixpkgs instead of having it in NixOS?
There was a problem hiding this comment.
Given my comment here: #27923 (comment) I'll expand:
My intention wasn't for any of the blocking packages to change. ie: tested remains completely unchanged. The major change is in not fanning out and building the whole package set.
If this isn't what we want that is okay, but to keep this the same, I think the code here does that correctly.
| @@ -2,7 +2,7 @@ | |||
| the load on Hydra when testing the `stdenv-updates' branch. */ | |||
|
|
|||
| { nixpkgs ? { outPath = (import ../../lib).cleanSource ../..; revCount = 1234; shortRev = "abcdef"; } | |||
| , supportedSystems ? [ "x86_64-linux" "i686-linux" "x86_64-darwin" ] | |||
| , supportedSystems ? [ "x86_64-linux" "x86_64-darwin" ] | |||
There was a problem hiding this comment.
Wasn't the plan to build the nixos-small channel for i686-linux but maybe remove it as constituents of the tested job?
There was a problem hiding this comment.
That isn't what I was expecting, but we could. I was expecting nixos-unstable to block on i686 but no -small's, and no wide package tree -- just essentials.
|
I generally agree with @fpletz concerns, but this is a step in the right direction and doesn't block us from refining this further (thinking more about aarch64, splitting etc.), so I'm merging this. |
| @@ -54,43 +57,36 @@ let | |||
| jobs.manual | |||
| jobs.lib-tests | |||
| jobs.stdenv.x86_64-linux | |||
| jobs.stdenv.i686-linux | |||
There was a problem hiding this comment.
I know I'm late, but I'd personally keep stdenv in there, as we still want to support some packages built by it (e.g. wine).
|
Hmm yes I'm pretty sure I meant to leave that. Unfortunately I can't submit
PRs for the next couple days.
…On Sun, Aug 6, 2017 at 3:37 AM Vladimír Čunát ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In pkgs/top-level/release.nix
<#27923 (comment)>:
> @@ -54,43 +57,36 @@ let
jobs.manual
jobs.lib-tests
jobs.stdenv.x86_64-linux
- jobs.stdenv.i686-linux
I know I'm late, but I'd personally keep stdenv in there, as we still want
to support *some* packages built by it (e.g. wine).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#27923 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAErrK5bV9e7VD06A1hu-O9fmYAzJC9aks5sVW02gaJpZM4OtNvm>
.
|
|
That doesn't matter; I'll push that change myself, but first I'll have a look why all |
The typo removed also all aarch64-linux on Hydra.
|
We might reconsider what to do about i686 nixos tests of big stuff. I now removed |
|
Actually, there's most likely no hurry for such actions, as since this PR the x86_64+i686 queue tends to be clearly the least loaded of all. (Darwin has the longest queue now, almost all the time.) |
|
Offtopic. But seems to be an appropriate place to ask this.
This constant lack of resources on hydra seems strange to me. It takes
<12 hours to rebuild every single package I use (and I use a lot) on a
4-core i5 with 32G of RAM (I build everything on tmpfs, makes a lot of
difference). When building on my little distributed cluster (16 cores)
its basically a matter of waiting for gcc to build, and then for webkit
and firefox to finish.
I do not rebuild every single commit, of course, but still.
What hardware official hydra runs on today?
Why can't hydra use distcc to speed up heavy builds?
Why can't it impurely cheat a little with ccache if it really wants to
build every single commit?
(Also, I think it would really help for nix to look at load averages of
the machines in the cluster, not just span `n` builders that do `make -j
n` when `enableParallelBuilding` is on. But that's offtopic.)
|
|
It doesn't build every commit, and Hydra is quite a strong set of machines now. (You can get an idea by looking at https://hydra.nixos.org/machines ) The largest problem here are changes in stdenv and other "mass-rebuild" packages. We have quite a lot of those nowadays, and doing one staging rebuild in two weeks would probably be an inconveniently slow feedback (for humans fixing the problems there). The worst part of the testing is that half of people isn't even able to build on darwin or aarch64 in any direct way. |
|
|
It doesn't build every commit, and Hydra is quite a strong set of machines now. (You can get an idea by looking at https://hydra.nixos.org/machines )
Ok, thanks.
The largest problem here are changes in stdenv and other "mass-rebuild" packages. We have quite a lot of those nowadays, and doing one staging rebuild in two weeks would probably be an inconveniently slow feedback (for humans fixing the problems there).
More the reason to seriously consider using ccache for staging, I think.
Also, I was wondering if some derivation preprocessing could possibly
help with that. Seems to me that many heavy "mass-rebuild"s are
refactorings of the infrastructure. Wouldn't it be nice if nix could
"partially evaluate" derivations (remove comments, normalize whitespace,
substitute non-recursive shell functions, etc) before assigning them a
hash. Most refactorings wouldn't produce any changes to derivations. One
could actually use that fact for self-assurance when refactoring
(refactor, nix-build, get the same output immediately, "Yay! I have not
f*cked up!").
The only problem I immediately see is that evaluation time for
derivations would go up.
The worst part of the testing is that half of people isn't even able to build on darwin or aarch64 in any direct way.
I agree. Also Travis.
- Load averages are considered by the `-l${NIX_BUILD_CORES}` option to make and equivalents. I'd personally also try a jobserver, as the current way is prone to fluctuations, but it doesn't seem like it could make a significant difference.
I'm not that sure. Last time I measured building two firefoxes in
parallel with `-j${NIX_BUILD_CORES}` I had more than 4x slowdown
(because too many threads compete for CPU caches, I think).
A lot of that was simply wasted because one would expect two firefoxes
to share a lot of code and so they should get a lot of ccache hits.
Another common nix artifact I see frequently:
* run one `nix-store --realise`, starts working as expected,
* run another one with the same derivation, it starts building something
too, I would expect it to wait with "no build slots".
(Also, "no build slots" stalls and waits, while "no nixbld users" produces an error.)
- `ccache` could help significantly, but (1) it's slightly risky for
purity (IIRC Eelco might oppose it)
I'm unsure about a good way to integrate ccache into stdenv myself. I
probably wouldn't trust it to build stdenv (at least, not by default),
and building tiny packages with it seems wasteful, but having an option
of using ccache at least for selected packages like geko and webkit
would be really nice, I think.
and in the current case of quite many build slaves it wouldn't help that much on Hydra, assuming the cache would be local (and distributed ccache would be quite a notrivial setup).
AFAIK its just works™ over NFS.
|
That is done – the
These cases always wait on my machine, even in case of using distributed builds. (iff the hashes are the same)
We have ccache over NFS could be good, but the build farm currently uses a mix of slaves that aren't co-located, so each compiler invocation would necessarily induce a noticeable latency. That might not be a problem by itself, but I expect it would be more fragile and we would probably want to increase the load, as much more waiting for network would be expected. We might even serve the ccache, similarly to current binary cache. (!!) |
|
That *is* done – the `*.drv` files are quite a lot canonicalized.
Not to any reasonable degree. I just checked. Adding a single empty line
to any shell script in `stdenv/generic` causes a mass rebuild.
I remember stopping myself from fixing typos in comments of expressions
I read (several times) because of the mass rebuild that caused.
With intensional store I expect we could save the propagation of mass rebuilds in case of bit-equal outputs, but that's a far-away dream...
It also wouldn't work in general for packages because many packages
refer to themselves by `/nix/store` path in their own output. In configs
and man pages, for example.
But we could probably do that for binaries.
Nix could link binaries (and other files too, put they are less
interesting for this discussion) from `/nix/store/<package>/bin` to e.g.
`/nix/store/bin/<hash of the binary>` and resolve
`{pkgs.package}/bin/<binary>` into the latter path instead.
Note that `nix-store --optimise` kinda does the first half already. All
what's left is the resolve and the ability to depend on files (not
packages) by hash. The problem is that nix derivations become dynamic
because you can't compute the output hash without having binaries of the
inputs and so `nix-instantiate` and `nix-store --realise` become a
single thing (or it needs to have evaluation passes or something).
That last part is a hard thing to swallow.
Another possibility that I entertained here is caching the final build
root of the derivation you build. If you trust the build system you
could just dump the results of the `unpackPhase` on top of the results
of the previous build of the same package and hope it will work. It
probably wouldn't, though.
So, I think, ccache is the way to go. Sure, caching more results would
be good, but I see no other realistic and more pure way to do it than
ccache.
By the way.
Another impure thing that would help a lot and could be absolutely
transparent is caching of source repositories. Like, say, fetchgit. Why
refetch it every time when you can just `git fetch` into a local cached
copy and `fetchgit` from it instead?
> Another common nix artifact I see frequently: * run one `nix-store --realise`, starts working as expected, * run another one with the same derivation, it starts building something too, I would expect it to wait with "no build slots".
These cases always wait on my machine, even in case of using distributed builds. (iff the hashes are the same)
Try it next time you rebuild the whole system. Build stdenv separately
first so that it won't block on it and then run those two realises.
I get that behavior even without distributed builds.
We have `ccacheStdenv`, but I have no idea if it has bitrotten or something.
Nah, it's fine. I just built it and it still passes all its tests with
gcc-6. By
> I'm unsure about a good way to integrate ccache into stdenv myself.
> ...
I mean
* sure, you can overrideCC like ccacheStdenv does (needs changes to
nix.conf too because sandbox, but it's tolerable),
* but you can only override stdenv for all packages or selected packages
and I would like to have "classes" of packages, like all
`requiredSystemFeatures = [ "big-parallel" ];` to be built with ccache
or something,
* and some work needs to be done to build stdenv using ccache and I'm
unsure about how it should be done. How the following plan sounds?
input: bootstrap-gcc
stages:
-> bootstrap-ccache: ccache built by bootstrap-gcc
-> stage1-gcc built by bootstrap-ccache with bootstrap-gcc
-> stage2-gcc built by bootstrap-ccache with stage1-gcc
-> stage2-ccache build by bootstrap-ccache with stage2-gcc
outputs: stdenv with stage2-gcc
ccacheStdenv with stage2-ccache with stage2-gcc
* some more work needs to be done to verify it produces same binaries as
the default stdenv and/or it should never be used for any results that
get into the binary cache. Because I'm paranoid that evil stuff can be
done with a ccache hash collision.
ccache over NFS could be good, but the build farm currently uses a mix of slaves that aren't co-located, so each compiler invocation would necessarily induce a noticeable latency. That might not be a problem by itself, but I expect it would be more fragile and we would probably want to increase the load, as much more waiting for network would be expected.
Well, yeah. Single ccache cache wouldn't work well with that. But you
could setup one cache per location and make hydra prefer putting
"similar" derivations (package name without version) to machines in the
"same" location (sharing ccache) when possible.
Or maybe there's some good hierarchical caching filesystem I don't know
about. Something like AFS, but hierarchical and with file locking.
*We might even serve the ccache, similarly to current binary cache. (!!)*
That could be cool, but the feasibility of it needs actual latency
measurements.
|
Yes, but given that the builder can be any program, nix can't assume any kind of semantics in general.
The intensional store proposal does count on self-references not changing the hashes. It's all in that old Eelco's thesis.
|
Motivation for this change
Only build a bare minimum of packages for i686 for 17.09 and beyond.
NixOS evaluation: https://hydra.nixos.org/jobset/nixos/grahamc-i686 (cuts out about 12,000 jobs)
Nixpkgs evaluation: https://hydra.nixos.org/jobset/nixpkgs/graham-i686 (cuts out about 21,000 jobs... is this right?)
NixOS still builds the same jobs in
testedto be sure we don't kill anyone's system, but we no longer span out and build everything.Nixpkgs won't build i686 at all, except for jobs where it is mandatory like Skype.
cc @FRidh @domenkozar @globin @edolstra
Related discussion on mailing list: https://groups.google.com/forum/#!topic/nix-devel/m7ikFjor24Y
see #27731
Things done
Please check what applies. Note that these are not hard requirements but merely serve as information for reviewers.
(nix.useSandbox on NixOS,
or option
build-use-sandboxinnix.confon non-NixOS)
nix-shell -p nox --run "nox-review wip"./result/bin/)