New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable parallel building for CMake based projects by default #32271
Conversation
f72577d
to
6e7dfa6
Compare
I have rebuilt everything again without any new failures. Disabling parallel building for Meson did not help Totem because it had it explicitly enabled; I disabled it for Totem and reenabled for Meson, with some approval from @jtojnar. |
This is funny: my attempt to disable parallel building of Totem failed because Ninja enables it unless explicitly disabled. All packages using Ninja have always been building in parallel, and those without |
46b7ed1
to
09c3362
Compare
Oh, I had noticed that large rebuilds were sometimes pushing load to a multiple of |
09c3362
to
4091c35
Compare
@GrahamcOfBorg eval |
94e8f5d
to
9d07d86
Compare
but keep it enabled by default
meson does not support parallel building of certain vala projects: mesonbuild/meson#1994 https://bugzilla.gnome.org/show_bug.cgi?id=784236 but at the moment this only affects gnome3.totem: ninja src/plugins/rotation/rotation.vapi [1/1] Compiling Vala source ../src/plugins/rotation/bacon-video.vapi ../src/plugins/rotation/totem-rotation-plugin.vala. FAILED: src/plugins/rotation/rotation@sha/totem-rotation-plugin.c src/plugins/rotation/rotation.h src/plugins/rotation/rotation.vapi valac -C --pkg clutter-gtk-1.0 --pkg cogl-pango-1.0 --pkg libpeas-1.0 --pkg gtk+-3.0 --color=always --directory src/plugins/rotation/rotation@sha --basedir ../src/plugins/rotation --library rotation --header src/plugins/rotation/rotation.h --vapi ../rotation.vapi --girdir=/tmp/nds-build-gnome3.totem/totem-3.26.0/build/src --pkg=Totem-1.0 ../src/plugins/rotation/bacon-video.vapi ../src/plugins/rotation/totem-rotation-plugin.vala error: Package `Totem-1.0' not found in specified Vala API directories or GObject-Introspection GIR directories
This is overridable by providing a custom build phase or setting dontUseNinjaBuild = true.
Upstream recommends Ninja and has a cmake script that does not support GNU Make: dino/dino#230
9d07d86
to
199b426
Compare
This is ready, but I will also describe the kinds of bugs in individual packages that I have encountered while working on this PR. |
I have identified two more cases. First, |
The issue of unfair distribution of CPU resources in favor of builders that do not respect load average can be solved by putting the processes of each build user into a separate cgroup. This may be possible with some existing cgroup manager, without changing nix-daemon. It should prevent timeouts in Go and QEMU tests. |
Motivation for this change
This obviously benefits small rebuilds. People contributing new packages, updating and fixing bugs in existing packages won't have to wait as much. Reviewers and maintainers of Nixpkgs won't have to ask for
enableParallelBuilding
, push it into pull requests, or bundle it when they commit something that rebuilds packages with parallel building not enabled.The downside of parallel building with make — unpredictable failures that are difficult to reproduce — is almost mitigated by build systems that automatically generate complete dependencies between build commands.
During the testing of this PR, I have encountered just 4 packages that failed (details are in the commits of the PR):
editorconfig-core-c
running doxygen multiple times from the same template with the same output directory has a race on directory creationshogun
running multiple ply.yacc that clobber temporary files of each otherdino
using (and recommending) Ninja and writing their cmake scripts in a way not supported by GNU Make generator of CMaketotem
relying on a feature missing from MesonI have reported the first three bugs upstream and disabled parallel building for them. The fourth was already reported both to Totem and Meson, with both sides agreeing that this is a bug in Meson, so I have disable parallel building for Meson. @jtojnar, since this seems to affect just one package, could you consider this issue and decide if we should rather enable parallel Meson and disable parallel Totem?
This PR also adds
-GNinja
to cmake when the build phase isninjaBuildPhase
, which is an obvious thing to do after #28444 has added the Ninja setup hook.Things done
build-use-sandbox
innix.conf
on non-NixOS)pkgs/top-level/release.nix
./result/bin/
)