Wait for build users when none are available #3564

balsoft · 2020-05-05T12:53:53Z

Descriptions from there:

It would be nice to have the option of waiting for a build user to become free instead of failing to build.
I'm talking about this error: error: all build users are currently in use; consider creating additional users and adding them to the ‘nixbld’ group
Is there ever a case where failing is the preferred option to waiting? Hopefully waiting can become the default behaviour.

Additional implementation details: when build users are available, the behavior is exactly the same as before (choose the first available user, take a lock on it, use it). However, when no build users are available at the moment, repeat the search every two seconds. Whenever a build user becomes available, choose it and continue as usual. Previous behavior was to error out.

Tested on:

NixOS
Non-NixOS Linux
Darwin
other platforms.

Note:
I have tried implementing a more "proper" solution with libfswatch (to only check the userpool when it changes), but for some reason the callback didn't trigger when lock files were released. If you believe that my work on implementing this with libfswatch is of interest, I can publish it.

domenkozar · 2020-05-05T13:50:49Z

Thank you a thousand times!

LnL7 · 2020-05-05T15:45:40Z

I think this also resolves the fact that we make 32 build users in the installer, this was mostly the reason for it.

balsoft · 2020-05-05T15:46:51Z

I think this also resolves the fact that we make 32 build users in the installer, this was mostly the reason for it.

This is my next nixpkgs PR, yes ;)

src/libstore/build.cc

edolstra · 2020-05-06T09:16:21Z

This is not a good solution because it puts the entire build loop to sleep (since it's single-threaded). So no other builds/substitutions will make any progress until another process releases a build user. In fact, it seems that this will deadlock if this process itself has acquired all build users (since then it can't make progress to release any build users).

For comparison, if we can't acquire a lock on an output path, the DerivationGoal is put to sleep so that other goals can make progress:

    if (!outputLocks.lockPaths(lockFiles, "", false)) {
        worker.waitForAWhile(shared_from_this());
        return;
    }

balsoft · 2020-05-06T09:18:36Z

@edolstra Thanks for the suggestion, I'll implement the same logic.

balsoft · 2020-05-08T09:28:02Z

Please, re-evaluate if possible.

I have changed the behaviour to match that of output locks. However, now #2069 is even more of an issue.

balsoft · 2020-05-08T09:33:54Z

My testing methodology, for anyone interested:

Run nix-daemon from this version of nix
Create a file
test-nix.nix

with import <nixpkgs> {};
let newPackage = n: hello.overrideAttrs (_: { name = "hello-${toString builtins.currentTime}-${toString n}"; });
in buildEnv { name = "env"; ignoreCollisions = true; paths = builtins.genList newPackage 100; }

nix build -f test-nix.nix --max-jobs $BUILDUSERS+3 (to test for deadlock mentioned above)
nix build -f test-nix.nix as many times as needed to run out of build users.

If the last two commands succeed and use up all available build users while not crashing and not using users two times over, my patch is correct.

I'd be very grateful if someone tested this and reported the results on non-NixOS linux and/or Mac.

balsoft · 2020-05-08T09:35:11Z

@domenkozar I had a temptation to do so here, but I think it merits a separate PR. I'll create one now.

Actually, I think it's slightly more complex than I anticipated. We need to monitor for changes of the reason for waiting.

balsoft · 2020-05-08T10:08:01Z

I'll come back to #2069 when I have time.

balsoft · 2020-05-08T16:21:20Z

@domenkozar fixed (architecturally) in #3577, will rebase that if/when this PR is merged.

edolstra · 2020-05-12T16:01:57Z

src/libstore/build.cc

@@ -530,6 +532,12 @@ class UserLock
 UserLock::UserLock()
 {
    assert(settings.buildUsersGroup != "");
+    createDirs(settings.nixStateDir + "/userpool");
+    /* Mark that user is not enabled by default */
+    uid = 0;


0 seems a risky choice as a sentinel value for UIDs given that 0 == root. So if there's ever any confusion, we might end up doing an operation as/to root. Might be better to use -1 or std::optional<uid_t>.

Alternatively, to ensure that UserLock always represents a valid user, have a builder function that returns std::unique_ptr<UserLock> so it can represent failure to acquire a user.

I've thought about adding a function that can represent a failure, and it does appear to be a better solution from some perspectives, but I didn't want to make the diff too big (since this is my first PR). Now that it's inevitably going to be massive, I'll wrap the result in optional and make it represent failure.

edolstra · 2020-05-12T16:07:12Z

src/libstore/build.cc

@@ -1391,6 +1394,30 @@ void DerivationGoal::tryToBuild()
 {
    trace("trying to build");

+    /* If `build-users-group' is not empty, then we have to build as
+       one of the members of that group. */


If I understand this correctly, it will try to acquire a build user even when we're doing a remote build, which is unnecessary. Maybe it's better to add a new state to the state machine, e.g. startBuild, with a transition from tryToBuild to startBuild if we're doing a local build.

Oh yes, this sounds very much like something I didn't even think about.

I'll try to wrap my head around this and implement it next weekend.

Thank you again!

Added a state and (hopefully) fixed this.

balsoft · 2020-05-14T14:05:51Z

@edolstra I've (hopefully) fixed both of issues you've found, please look at it again when you have time.

The CI seems to be slightly broken (it can't find a builder for macOS).

I've tested this change on my NixOS desktop as explained above.

edolstra · 2020-05-18T11:56:47Z

Thanks!

osa1 · 2021-06-23T14:20:33Z

Sorry for reviving and old thread and n00b question, but, what does it mean if I'm still getting "all build users are currently in use" errors today? Does that mean my nix setup is too old and does not have this patch? Or could it be a configuration issue?

balsoft · 2021-06-23T14:22:32Z

This patch never became part of the 2.3 (stable) branch, so it's only available on master/2.4pre right now. @edolstra is there any chance to backport this to 2.3?

osa1 · 2021-06-23T14:31:02Z

Thanks for the quick response. Do we have a public URL for 2.4pre that I can use? I'm using https://github.com/cachix/install-nix-action which allows specifying install URL (default is https://nixos.org/nix/install).

balsoft · 2021-06-23T19:21:41Z

Please see https://github.com/cachix/install-nix-action#usage-with-flakes (except you don't need all the experimental-features bits, just copy install_url and install_options)

This version should include NixOS/nix#3564 and allow us to increase max-jobs. See #2621 for the previous attempt, reverted in 8365e73. The install URL is obtained from https://github.com/NixOS/nix/runs/2895674848 (NixOS/nix@323e545)

Wait for build users when none are available

04967de

cole-h reviewed May 5, 2020

View reviewed changes

src/libstore/build.cc Outdated Show resolved Hide resolved

Don't block while waiting for build users

14073fb

Mention build users in the 'waiting for' message

772e5db

domenkozar requested a review from edolstra May 11, 2020 16:15

edolstra requested changes May 12, 2020

View reviewed changes

Don't lock a user while doing remote builds

183dd28

edolstra merged commit 0ed946a into NixOS:master May 18, 2020

domenkozar mentioned this pull request Jun 11, 2020

Wait for build users instead of failing #1216

Closed

balsoft mentioned this pull request Jun 16, 2020

New nixpkgs committers requests NixOS/nixpkgs#50105

Open

B4dM4n mentioned this pull request Jun 29, 2020

max-jobs option is ignored #3763

Closed

osa1 mentioned this pull request Jun 23, 2021

Increase parallel jobs on GitHub CI dfinity/motoko#2621

Closed

osa1 mentioned this pull request Jun 24, 2021

GH CI: update nix, increase max-jobs dfinity/motoko#2624

Merged

balsoft deleted the wait-for-builders branch June 24, 2021 12:47

nomeata mentioned this pull request Aug 13, 2021

--daemon-user-count 4? cachix/install-nix-action#79

Closed

fricklerhandwerk mentioned this pull request Sep 13, 2022

Hard not to run out of build users when using --check #1804

Closed

flokli mentioned this pull request Oct 31, 2023

[2.3-maintenance] Wait for build users when none are available #9261

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait for build users when none are available #3564

Wait for build users when none are available #3564

balsoft commented May 5, 2020

domenkozar commented May 5, 2020

LnL7 commented May 5, 2020

balsoft commented May 5, 2020

edolstra commented May 6, 2020

balsoft commented May 6, 2020

balsoft commented May 8, 2020

balsoft commented May 8, 2020 •

edited

balsoft commented May 8, 2020 •

edited

balsoft commented May 8, 2020

balsoft commented May 8, 2020 •

edited

edolstra May 12, 2020

edolstra May 12, 2020

balsoft May 12, 2020

edolstra May 12, 2020

balsoft May 12, 2020

balsoft May 14, 2020

balsoft commented May 14, 2020 •

edited

edolstra commented May 18, 2020

osa1 commented Jun 23, 2021

balsoft commented Jun 23, 2021

osa1 commented Jun 23, 2021

balsoft commented Jun 23, 2021 •

edited

Wait for build users when none are available #3564

Wait for build users when none are available #3564

Conversation

balsoft commented May 5, 2020

domenkozar commented May 5, 2020

LnL7 commented May 5, 2020

balsoft commented May 5, 2020

edolstra commented May 6, 2020

balsoft commented May 6, 2020

balsoft commented May 8, 2020

balsoft commented May 8, 2020 • edited

balsoft commented May 8, 2020 • edited

balsoft commented May 8, 2020

balsoft commented May 8, 2020 • edited

edolstra May 12, 2020

Choose a reason for hiding this comment

edolstra May 12, 2020

Choose a reason for hiding this comment

balsoft May 12, 2020

Choose a reason for hiding this comment

edolstra May 12, 2020

Choose a reason for hiding this comment

balsoft May 12, 2020

Choose a reason for hiding this comment

balsoft May 14, 2020

Choose a reason for hiding this comment

balsoft commented May 14, 2020 • edited

edolstra commented May 18, 2020

osa1 commented Jun 23, 2021

balsoft commented Jun 23, 2021

osa1 commented Jun 23, 2021

balsoft commented Jun 23, 2021 • edited

balsoft commented May 8, 2020 •

edited

balsoft commented May 8, 2020 •

edited

balsoft commented May 8, 2020 •

edited

balsoft commented May 14, 2020 •

edited

balsoft commented Jun 23, 2021 •

edited