Substitute drv outputs #4238

thufschmitt · 2020-11-09T15:49:45Z

~~Depends on #4225, only commits after 97aca21 are new.~~

Depends on #4487

Changes the build loop to allow substituting derivation outputs.

roberth · 2020-11-10T12:55:25Z

src/libstore/daemon.cc

+        logger->startWork();
+        auto outputId = DrvOutputId::parse(readString(from));
+        auto outputPath = StorePath(readString(from));
+        store->registerDrvOutput(outputId, DrvOutputInfo{outputPath});


Shouldn't this require trusted user or signature verification?

It definitely should, yes, and that's one of the necessary follow-ups.

I think it's okay to keep it for a subsequent PR though, as long as it is behind the ca-derivations feature flag (which it isn't atm, I'll have to add it)

Where do you track these problems?

Curently mostly on a sheet of paper on my desk. Opening proper issues is still on my todo-list

I've fixed it in 3be2780 (as part of #4225). Now registerDrvOutput requires the experimental feature ca-derivations to be enabled if we can't prove that it's indeed a valid output of an input-addressed derivation.

I've also opened #4248 to track the signature problem

dpulls · 2021-01-28T10:06:19Z

🎉 All dependencies have been resolved !

Change the `nix-build` logic for linking/printing the output paths to allow for some outputs to be missing. This might happen when the toplevel derivation didn't have to be built, either because all the required outputs were already there, or because they have all been substituted.

Simple test to ensure that `nix-build && nix-collect-garbage && nix-build -j0` works as it should

Once a build is done, get back to the original derivation, and register all the newly built outputs for this derivation. This allows Nix to work properly with derivations that don't have all their build inputs available − thus allowing garbage collection and (once it's implemented) binary substitution

Where a `RealisedPath` is a store path with its history, meaning either an opaque path for stuff that has been directly added to the store, or a `Realisation` for stuff that has been built by a derivation This is a low-level refactoring that doesn't bring anything by itself (except a few dozen extra lines of code :/ ), but raising the abstraction level a bit is important on a number of levels: - Commands like `nix build` have to query for the realisations after the build is finished which is fragile (see 27905f1 for example). Having them oprate directly at the realisation level would avoid that - Others like `nix copy` currently operate directly on (built) store paths, but need a bit more information as they will need to register the realisations on the remote side

That way we can copy the realisations too (in addition to the store paths themselves)

To prepare for the upcoming DrvOutputSubstitutionGoal

Requires a slight update to the test infra to work properly

Ericson2314 · 2021-01-28T17:06:15Z

@regnat I don't think this depends on anything anymore?

Also, there is another place we want to substitute resolutions: when resolving CA derivations. In that case we don't want to fetch the data. That means we would need to not have resolution goals spawn path goals, at least not unconditionally.

Ericson2314 · 2021-01-28T17:22:09Z

Getting into the weeds, I think some of the inputsRealised logic on resolving would need to be factored out so it can also be used twice`.

Basically we end up with:

Try to fetch the outputs.
Try to resolve the inputs. If we can, then resolve the derivation, and delegate to building that (call bits factored out from inputsRealised call)
Try to download the inputs (same)
Resolve the derivation (inputsRealised, and same logic, but now calls factored-out bits to do it)

This can be done in a follow-up PR, of course.

thufschmitt · 2021-01-29T14:31:39Z

@regnat I don't think this depends on anything anymore?

It still depends #4487 which itself depends on #4340 and #4372 (unless we want to merge it into #4487. I don't really care about the order as long as everything ends-up merged)

Also, there is another place we want to substitute resolutions: when resolving CA derivations. In that case we don't want to fetch the data. That means we would need to not have resolution goals spawn path goals, at least not unconditionally.
Getting into the weeds, I think some of the inputsRealised logic on resolving would need to be factored out so it can also be used twice`.

Fully agree. I've opened #4493 to track that (sorry, I though there was already an issue about it)

dpulls · 2021-02-26T11:20:44Z

🎉 All dependencies have been resolved !

roberth reviewed Nov 10, 2020

View reviewed changes

thufschmitt force-pushed the substitute-drv-outputs branch 2 times, most recently from a9090ae to b2a19e0 Compare November 12, 2020 12:15

thufschmitt added the ca-derivations Derivations with content addressed outputs label Nov 16, 2020

thufschmitt force-pushed the substitute-drv-outputs branch from b2a19e0 to 45c0a9b Compare November 17, 2020 09:51

thufschmitt mentioned this pull request Nov 17, 2020

Track the dependencies of derivation outputs #4267

Closed

thufschmitt force-pushed the substitute-drv-outputs branch 3 times, most recently from 20a3ccc to 4f0541e Compare November 18, 2020 09:38

thufschmitt requested a review from edolstra November 18, 2020 09:39

thufschmitt force-pushed the substitute-drv-outputs branch 4 times, most recently from 9e86318 to 081b0bf Compare November 24, 2020 10:23

thufschmitt force-pushed the substitute-drv-outputs branch 3 times, most recently from 661b315 to e10741c Compare December 1, 2020 16:23

thufschmitt added this to the ca-derivations-mvp milestone Dec 11, 2020

thufschmitt force-pushed the substitute-drv-outputs branch 2 times, most recently from ddc348a to 0d77839 Compare January 28, 2021 16:34

thufschmitt added 9 commits January 28, 2021 17:43

Test the garbage collection of CA derivations

5f23f9b

Simple test to ensure that `nix-build && nix-collect-garbage && nix-build -j0` works as it should

Merge branch ca/store-unresolved-outputs into ca/copy-drv-outputs

1494be0

Merge branch ca/drvoutputs-commands into ca/copy-drv-outputs

671b13e

Use RealisedPaths in copyPaths

5082f4f

That way we can copy the realisations too (in addition to the store paths themselves)

Add a test for the copy of CA paths

a41f350

Test more cases of copy-ing ca derivations around

f420043

thufschmitt changed the base branch from master to ca/copy-drv-outputs January 28, 2021 16:46

thufschmitt added 4 commits January 28, 2021 17:47

SubstitutionGoal -> PathSubstitutionGoal

a0bbf39

To prepare for the upcoming DrvOutputSubstitutionGoal

Allow substituting drv outputs when building

4075dba

Move the CA tests to a sub-directory

cf6eab8

Requires a slight update to the test infra to work properly

Add a test for the remote caching of CA derivations

1c6c71d

thufschmitt force-pushed the substitute-drv-outputs branch from 0cb72be to 1c6c71d Compare January 28, 2021 16:48

thufschmitt mentioned this pull request Jan 29, 2021

Substitute the realisations from the remote caches #4493

Open

thufschmitt force-pushed the ca/copy-drv-outputs branch 3 times, most recently from abf404b to 8ac4133 Compare February 19, 2021 14:50

thufschmitt force-pushed the ca/copy-drv-outputs branch from 75bd078 to c43f446 Compare February 25, 2021 16:32

edolstra closed this Feb 26, 2021

edolstra deleted the branch NixOS:ca/copy-drv-outputs February 26, 2021 11:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Substitute drv outputs #4238

Substitute drv outputs #4238

thufschmitt commented Nov 9, 2020 •

edited

roberth Nov 10, 2020

thufschmitt Nov 10, 2020

roberth Nov 11, 2020

thufschmitt Nov 11, 2020

thufschmitt Nov 12, 2020

dpulls bot commented Jan 28, 2021

Ericson2314 commented Jan 28, 2021

Ericson2314 commented Jan 28, 2021

thufschmitt commented Jan 29, 2021 •

edited

dpulls bot commented Feb 26, 2021

Substitute drv outputs #4238

Substitute drv outputs #4238

Conversation

thufschmitt commented Nov 9, 2020 • edited

roberth Nov 10, 2020

Choose a reason for hiding this comment

thufschmitt Nov 10, 2020

Choose a reason for hiding this comment

roberth Nov 11, 2020

Choose a reason for hiding this comment

thufschmitt Nov 11, 2020

Choose a reason for hiding this comment

thufschmitt Nov 12, 2020

Choose a reason for hiding this comment

dpulls bot commented Jan 28, 2021

Ericson2314 commented Jan 28, 2021

Ericson2314 commented Jan 28, 2021

thufschmitt commented Jan 29, 2021 • edited

dpulls bot commented Feb 26, 2021

thufschmitt commented Nov 9, 2020 •

edited

thufschmitt commented Jan 29, 2021 •

edited