Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always keep the CA derivations around #4139

Closed
wants to merge 1 commit into from

Conversation

thufschmitt
Copy link
Member

Workaround for #4138

@edolstra
Copy link
Member

Does this mean that CA derivations are never garbage-collected?

@thufschmitt
Copy link
Member Author

They aren't until their output-path is (like if keep-derivations is set to true in the config)

@Ericson2314
Copy link
Member

In fact, perhaps they should be kept around even if their output isn't.

Suppose we have:

( resolved-gcc.drv , "out" ) -> foo
( resolved-bash.drv [ foo / "gcc" ] , "out" ) -> bar

If we get rid of the row for foo, then if someone tries to rebuild (unresolved) bash.drv they'll end up rebuilding GCC!

We're going to need a pretty big migration I'm afraid to have the bar row preserve the foo row without also preserving the foo path itself.

Copy link
Member

@Ericson2314 Ericson2314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All that said, this PR makes the situation wholly better, and it is an easy enough stop-gap to back out of whenever we do do the sqlite overhaul.

@thufschmitt
Copy link
Member Author

In fact, perhaps they should be kept around even if their output isn't.

Suppose we have:

( resolved-gcc.drv , "out" ) -> foo
( resolved-bash.drv [ foo / "gcc" ] , "out" ) -> bar

If we get rid of the row for foo, then if someone tries to rebuild (unresolved) bash.drv they'll end up rebuilding GCC!

We're going to need a pretty big migration I'm afraid to have the bar row preserve the foo row without also preserving the foo path itself.

I'm not 100% sure, but I think that the DerivationOutputs entries stay there as long as the corresponding derivation is there (even if their output is gc-ed), so in that case the first mapping would stay alive because resolved-gcc.drv would (as it's a dependency of resolve-bash.drv)

@Ericson2314
Copy link
Member

Ah, but resolved-gcc.drv is not a dependency of resolve-bash.drv, because resolved floating CA derivations always have empty inputDrvs. That's the difficulty.

@Ericson2314
Copy link
Member

Ah CI failed because of that annoying spurious failure. Do we have an issue for it?

@thufschmitt
Copy link
Member Author

Ah, but resolved-gcc.drv is not a dependency of resolve-bash.drv, because resolved floating CA derivations always have empty inputDrvs. That's the difficulty.

Oh yes because we only have this mapping for resolved drvs 🤔
I think we need to dig all that a little to properly fix this (but the workaround is worth keeping in the meantime)

@edolstra
Copy link
Member

In fact, perhaps they should be kept around even if their output isn't.

But then derivations will accumulate forever... Or they could be treated like a cache and have some expiration policy.

@edolstra edolstra added the ca-derivations Derivations with content addressed outputs label Oct 30, 2020
@thufschmitt thufschmitt added this to the ca-derivations-mvp milestone Dec 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ca-derivations Derivations with content addressed outputs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants