New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Track the dependencies of derivation outputs #4267
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thufschmitt
force-pushed
the
track-drvoutput-dependencies
branch
4 times, most recently
from
November 18, 2020 09:38
559bad1
to
0b960c8
Compare
thufschmitt
force-pushed
the
track-drvoutput-dependencies
branch
4 times, most recently
from
November 24, 2020 10:23
c4ae03e
to
52b6357
Compare
thufschmitt
force-pushed
the
track-drvoutput-dependencies
branch
2 times, most recently
from
November 27, 2020 20:17
5fc194c
to
04177fd
Compare
…sed ones Fix an overlook of NixOS#4056
This requires adding `nix` to its own closure which is a bit unfortunate, but as it is optional (the test will be disabled if `OUTER_NIX` is unset) it shouldn't be too much of an issue. (Ideally this should go in another derivation so that we can build Nix and run the test independently, but as the tests are running in the same derivation as the build it's a bit complicated to do so).
For each “known” derivation output, store: - its output id - its output path This comes with a set of needed changes: - New `drv-output-info` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryDrvOutputInfo` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryDrvOutputInfo`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
Copying a derivation output is equivalent to copy the output path, except that it will also link the output to the path (with `linkDeriverToPath` on the remote store)
Commands that accept store paths or installables as input can now take a derivation path with outputs. Like when the input is a nix expr, the derivation will be realised if needed.
Generalises `StorePathsCommand` to allow manipulating `Buildables` rather than just store paths.
Mostly dummy atm as it just computes the closure of the output path, but might be extended later
Add a new table for tracking the derivation output mappings. We used to hijack the `DerivationOutputs` table for that, but (despite its name), it isn't a really good fit: - Its entries depend on the drv being a valid path, making it play badly with garbage collection and preventing us to copy a drv output without copying the whole drv closure too; - It dosen't guaranty that the output path exists; By using a different table, we can experiment with a different schema better suited for tracking the output mappings of CA derivations. (incidentally, this also fixes NixOS#4138)
That way we can `nix copy --from` a store that doesn't have the derivation but just its outputs
That way we don't need to resolve it again to know its output paths. In particular, this means that we don't need to keep its whole build-time closure
`buildPaths` can be called even for stores where it's not defined in case it's bound to be a no-op. The “no-op detection” mechanism was only detecting the case wher `buildPaths` was called on a set of (non-drv) paths that were already present on the store. This commit extends this mechanism to also detect the case where `buildPaths` is called on a set of derivation outputs which are already built on the store.
To prepare for the upcoming DrvOutputSubstitutionGoal
Requires a slight update to the test infra to work properly
Currently never used, nor set but will be useful shortly
We need to topologically sort them like we do for the store paths, otherwise the registration will fail because of an incomplete closure
thufschmitt
force-pushed
the
track-drvoutput-dependencies
branch
from
December 1, 2020 16:24
04177fd
to
8b8e75b
Compare
Closing in favor of #4836 as this one is severely outdated |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Builds on top of #4238
Fix #4249 by keeping track of all the dependencies of each derivation output.
The dependencies of a drv output
foo!out
is defined as the subset of its build inputs that are references of its output path − or said otherwise the minimal set of drv outputs that need to be realised to get the runtime closure of the output path (the motivation for this is given in #4238)