Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly store the outputs of CA derivations #4315

Closed
wants to merge 2 commits into from

Conversation

thufschmitt
Copy link
Member

For each “known” derivation output, store:

  • its output id
  • its resolved derivation
  • its output path

(morally what we want is two families of mappings: One mapping
unresolved derivations to their resolved version and one mapping
resolved derivations to their output path, but it's both simpler and
more efficient to combine them)

This comes with a set of needed changes:

  • New drv-output-info module declaring the types needed for describing
    these mappings
  • New Store::registerDrvOutput method registering all the needed informations
    about a derivation output (also replaces LocalStore::linkDeriverToPath)
  • new Store::queryDrvOutputInfo method to retrieve the informations for a
    derivations

This introcudes some redundancy on the remote-store side between
wopQueryDerivationOutputMap and wopQueryDrvOutputInfo.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where wopQueryDerivationOutputMap is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.

For the local store, this uses a different db tables that will only be added to the schema if the ca-derivations experimental flag is on, meaning that people not using it won't be affected by the schema change

@thufschmitt
Copy link
Member Author

I've reworked this a little to

  1. Remove the resolvedDrv field as it might not give much
  2. Remove some unused code
  3. Rename the added table into Realisations as OutputMappings was a rather ugly and imprecise term

@thufschmitt thufschmitt added the ca-derivations Derivations with content addressed outputs label Dec 8, 2020
@thufschmitt thufschmitt force-pushed the ca/properly-store-outputs branch 4 times, most recently from 3051b84 to dfeccb1 Compare December 8, 2020 09:46
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryDrvOutputInfo` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryDrvOutputInfo`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes NixOS#4138)
@thufschmitt
Copy link
Member Author

Rebased on top of latest master to fix a conflict

@thufschmitt
Copy link
Member Author

Closing in favor of #4330

@thufschmitt thufschmitt closed this Dec 8, 2020
@thufschmitt thufschmitt added this to the ca-derivations-mvp milestone Dec 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ca-derivations Derivations with content addressed outputs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant