[RFC 0014] Improve import from derivation #14
Conversation
|
@Ericson2314 did you get my review by email? Let me know what I can do to help. |
| There's currently a bug with `nix-repl` where if computation is interrupted (say with `SIGINT`), and the same computation is begun again, one will get an infinite recursion error despite there being no infinite recursion. | ||
| This is because the thunks stay black-holed—the indication that they are currently evaluated—despite the computation being aborted. | ||
| To fix this, we want aborted computations to instead unwind their stack, un-black-holing any thunks back the way they were before. | ||
| Mathematically, black-hole unwinding ensures that evaluation is idempotent. |
There was a problem hiding this comment.
Just a thought: this seems useful on it's own. Maybe split it in it's own RFC, I suspect smaller RFCs are more likely to make progress. Or just implement it?
There was a problem hiding this comment.
Seeing that that's just a bug fix, I might just open an issue. That issue can be linked from any tracking the RFC as a whole if it is accepted.
There was a problem hiding this comment.
Opened NixOS/nix#1407
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| Overlap with recursive Nix. |
There was a problem hiding this comment.
Do you have a link to what recursive Nix is? Just so the reader can compare.
|
The technical writing could be a lot better, but I get the gist of it. It appears that this is just an instance of "recursive make considered harmful", though. My longterm concern is that this will lead to a new workflow where people will start making use of the feature, which means that the system as a whole becomes dependent on more dynamic features while they could also be implemented with more static features (as it is now). In short, I would say this is a controversial feature. It's not like I can stop you, but I probably wouldn't add this feature if I had complete control over the project. |
|
Yes, dynamism should be the escape the escape hatch of last resort. I hope the burden of writing concrete syntax with an external program is enough to dissuade anyone from using it more. |
|
I should edit this RFC to also mention readFile. ReadFile of a derivation needs analogous improvements. |
|
So if I understand correctly, to summarize:
|
|
@domenkozar do you mean " start 1) again..."? If so yes. It could also just let |
|
@Ericson2314 We're doing triage on the RFC backlog, are you still interested in pursuing this RFC? The issue I see with this RFC is that it doesn't really work given Hydra's current design, specifically, the strict split between the evaluator and the queue runner. The evaluator is supposed to add rows to the This could be solved by having an integrated evaluator/queue runner, but then we don't need any partial evaluation tricks: we can just build IFDs as normal build steps during evaluation. (The only issue is that during evaluation there is no Build yet to account the BuildStep to, but that can be hacked around by having a temporary Build or whatever.) The downside of integrating the evaluator and queue runner is that it reduces security a bit, since (for example) the jobset Nix expression could try to read SSH credentials of the queue runner during evaluation using |
|
I'm going to close for the time being. I need to fix a few details, and then I have a new idea using derivations that build derivation trees to do this better still. |
|
@edolstra Also, thank you for that description of the hydra status quo. I'll try to better separate the Nix itself and hydra concerns from this PR, but on the hydra side I'm thinking that top-level derivations must not be auto-generated, but their dependencies may be. My new idea already supported that on the Nix side, and from your description, that would be allow Hydra to keep its existing division of labor. [If we had to refactor hydra that much, I'd declare it "totalled", i.e. it would be better to just use a rewrite that would also fix a slew of other issues.] |
|
In Nix, you can't have a .drv file without knowing all its dependencies. This is independent from any evaluation issues. |
|
@edolstra yes this would be a recursive Nix thing, where drv1 would build drv2 (and drv2's transitive dependencies' drvs). drv1 would be delayed evaluation of nix expressions, which we may or may not wish to special-case purely. That special case would purely be an optimization. |
* 166: Nix formatting Create 0101-nix-formatting.md WIP Go through a large part and agree on it Co-Authored-By: @piegamesde Update 0101-nix-formatting.md Rework a lot of things Update 0101-nix-formatting.md Move around some sections Reword the detailed section Minor updates Slight header changes again Updates Update 0101-nix-formatting.md Update after today's meeting Update 0101-nix-formatting.md Further updates in the meeting Update 0101-nix-formatting.md After todays meeting Update after meeting Rename to probably the right number Only use anchor links Improvements and additions - The sub-expression rule is now reworded and its own section with examples and rationale - Line length limit is now specified as we agreed-upon in the meeting - The operator section is rewritten to align more with the consensus Redo and explain operator special case Also remove the special case for non-chainable operators, barely any benefit in Nixpkgs * Operator chains outside bindings can also have a compact form * Make the operator compact form specific to binders * Fix accidentally formatted semicolons in alternatives * Minor changes * Light copy editing * Fix .git-blame-ignore-revs * Improve assert/with wording * Be more flexible with single-line element count * binder -> binding * unindent inherit semicolon, reshuffle binding/inherit sections (#14) * unindent inherit semicolon, reshuffle binding/inherit sections * fixup! Stuff * Give alternatives to `in` formatting * Expand on line break preservation * Add editorconfig * Expand argumentation against leading commas * Add @dasJ to the formatter team * Add shepherd team Co-authored-by: Linus Heckemann <git@sphalerite.org> * Various improvements (#15) * Various improvements - Remove unnecessary **Description** headers - Rename **Rationale and Alternatives** to just **Alternatives** - Insert must/may/should more diligently - Add some TODOs where things are unclear - Remove numberings from examples when not needed - Minor clarity improvements and simplifications throughout * Apply suggestions from code review Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu> --------- Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu> * Address TODOs and rework with/assert * Minor adjustments * Mention formatting Nix code in documentation * Working towards finalization (#16) - Defined absorption and absorbable terms - Adapted the existing RFC text to make use of these definitions, resulting in simplications of the text in many cases. - Updated `with` section to match the implementation - Updated the function declaration section to match the implementation - Sometimes, the function body may get absorbed - This used to be a special case scoped to bindings only, so it got removed there - Updated the operators section to match the implementation - Specify the format of non-chainable operators (somehow those got lost in the past) - Reworked bindings section. It should now be clear and specific enough. - Minor wording fixes * String section * Specify assert conditions * More absorption for multi-line arguments * How to update the standard format * Fix minor typos * Less lines for common function call patterns * Specify comments * Specify that the formatter should be as pure as possible With some exceptions * nit: fix list concatenation example (#17) * Update rfcs/0166-nix-formatting.md Co-authored-by: Doron Behar <doron.behar@gmail.com> * Add good indentation examples (#18) * Add another chainable operators example * justify difference in semicolon placement * Allow different parenthesized argument style * Clarify non-vertical alignment rule * Improved clarity of bindings rule * Improve bindings semicolon alternatives section --------- Co-authored-by: Silvan Mosberger <github@infinisil.com> Co-authored-by: Silvan Mosberger <silvan.mosberger@tweag.io> Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu> Co-authored-by: Yuriy Taraday <yorik.sar@gmail.com> Co-authored-by: Linus Heckemann <git@sphalerite.org> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Doron Behar <doron.behar@gmail.com>
New description
Build on #40 to allow pausing and resuming evaluation with derivations, such that we always have derivations for are end goals. This allows
hydraand variousnix-envqueries to work with their current architecture. Therefore, if this RFC is implemented,hydra.nixos.orgshould allow importing from derivations.Original description
Fix some issues with importing from derivations so it is nicer to use both manually and with CI. If this RFC is implemented,
hydra.nixos.orgshould allow importing from derivations.Rendered