Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track the dependencies of derivation outputs #4267

Closed
wants to merge 30 commits into from

Conversation

thufschmitt
Copy link
Member

Builds on top of #4238

Fix #4249 by keeping track of all the dependencies of each derivation output.

The dependencies of a drv output foo!out is defined as the subset of its build inputs that are references of its output path − or said otherwise the minimal set of drv outputs that need to be realised to get the runtime closure of the output path (the motivation for this is given in #4238)

@thufschmitt thufschmitt force-pushed the track-drvoutput-dependencies branch 4 times, most recently from 559bad1 to 0b960c8 Compare November 18, 2020 09:38
@thufschmitt thufschmitt added the ca-derivations Derivations with content addressed outputs label Nov 18, 2020
@thufschmitt thufschmitt force-pushed the track-drvoutput-dependencies branch 4 times, most recently from c4ae03e to 52b6357 Compare November 24, 2020 10:23
@thufschmitt thufschmitt force-pushed the track-drvoutput-dependencies branch 2 times, most recently from 5fc194c to 04177fd Compare November 27, 2020 20:17
This requires adding `nix` to its own closure which is a bit unfortunate,
but as it is optional (the test will be disabled if `OUTER_NIX` is unset) it
shouldn't be too much of an issue.

(Ideally this should go in another derivation so that we can build Nix and run
the test independently, but as the tests are running in the same derivation
as the build it's a bit complicated to do so).
For each “known” derivation output, store:
- its output id
- its output path

This comes with a set of needed changes:

- New `drv-output-info` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryDrvOutputInfo` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryDrvOutputInfo`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
Copying a derivation output is equivalent to copy the output path,
except that it will also link the output to the path (with
`linkDeriverToPath` on the remote store)
Commands that accept store paths or installables as input can now take a
derivation path with outputs. Like when the input is a nix expr, the
derivation will be realised if needed.
Generalises `StorePathsCommand` to allow manipulating `Buildables`
rather than just store paths.
Mostly dummy atm as it just computes the closure of the output path, but
might be extended later
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes NixOS#4138)
That way we can `nix copy --from` a store that doesn't have the derivation but
just its outputs
That way we don't need to resolve it again to know its output paths.
In particular, this means that we don't need to keep its whole build-time
closure
`buildPaths` can be called even for stores where it's not defined in case it's
bound to be a no-op.
The “no-op detection” mechanism was only detecting the case wher `buildPaths`
was called on a set of (non-drv) paths that were already present on the store.

This commit extends this mechanism to also detect the case where `buildPaths`
is called on a set of derivation outputs which are already built on the store.
@thufschmitt
Copy link
Member Author

Closing in favor of #4836 as this one is severely outdated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ca-derivations Derivations with content addressed outputs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Track the dependencies of derivation outputs
1 participant