New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stdenv: perform checks only when evaluating .drv and .out #33057
Conversation
11d2349
to
7f26e84
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The effect of this PR is good, but the implementation seems a bit too complex.
lib/customisation.nix
Outdated
addPassthru = drv: passthru: | ||
the derivation itself and then map all outputs with given | ||
functions. */ | ||
extendDerivation = passthru: f: d: o: drv: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to the name of the function, drv
should be the first parameter.
None of the f
, d
, o
use the first of the two arguments this function passes to them, so it should not pass it at all. They also simply return their second argument (what else can be done with an output path?), so it should not be passed either. The new function needs just one new argument isValid
(or { isValid ? true; }
) to use as:
drvPath = assert isValid; drv.${outputName}.drvPath;
outPath = assert isValid; drv.${outputName}.outPath;
7f26e84
to
941fd42
Compare
According to the name of the function, `drv` should be the first parameter.
I disagree. I didn't change the order of args of `addPassthru` because I didn't want to patch all the uses (I actually did that in the first version in nixos/related-packages-v3 branch, but then I got tired rebasing on every change of impl), but I think its `addPassthru` is wrong here. Reason: functions that modify data structures take structures as the last argument (because you commonly have to curry them), e.g. in Haskell:
```
$ ghci
λ> :m Data.Map
λ> :t Data.Map.insert
Data.Map.insert :: Ord k => k -> a -> Map k a -> Map k a
λ> :t Prelude.map
Prelude.map :: (a -> b) -> [a] -> [b]
```
I think, you argument here should be "you need to fix the order of arguments of `addPassthru`". I could do that later, if enough people care.
But I certainly don't want to introduce new functions with the wrong order of arguments.
None of the `f`, `d`, `o` use the first of the two arguments this function passes to them, so it should not pass it at all. They also simply return their second argument (what else can be done with an output path?), so it should not be passed either.
Correct. In the current use-case all that generality is not needed, but I can imagine use cases when it would be needed (e.g. some outputs having more checks, like, say, accessing "debug" output would check `CC_FLAGS` has "-O0" or something).
In general, I modeled `extendDerivation` after `mapAttrs`-family of functions, which is why it takes `k: v:`-style arguments.
What is really needed here, I think, is the top level expression wrapped in `f null` for consistency. Fixed, updated the branch.
The new function needs just one new argument `isValid` (or `{ isValid ? true; }`) to use as:
```
drvPath = assert isValid; drv.${outputName}.drvPath;
outPath = assert isValid; drv.${outputName}.outPath;
```
Correct. For this use case. But I'd rather add a function that covers all the use cases I can come up with, than introduce copy-paste later (`customization.nix` has copious amounts of copy-paste already, just look at the code there, like all the functions that modify outputs follow the same pattern, I would just generalize-rewrite half the functions there if I weren't sure it would cause the usual amounts of bikeshedding).
|
941fd42
to
cf6aa22
Compare
In general, I modeled `extendDerivation` after `mapAttrs`-family of functions, which is why it takes `k: v:`-style arguments.
Hmm. Actually, thinking some more about it, maybe I do need to change the order, since `extendDerivation` is somewhat like `fold` too, so `passthru` arg needs to be moved to the second to last. Done.
|
This is true of basic functions in functional programming, but easy currying is not the ultimate merit of a function. Agreement of the name with the order of arguments is more important. Moreover, often currying is more confusing than useful, especially if the number of arguments is large, such as in your 5 argument version of
You don't have to choose between these two alternatives. It is bad to copy and paste (not always), but it is also bad to support imaginary use cases at the cost of additional complexity of mysterious features that make one question "what is this for? where is it used? oh, it is not?". It is better to abstract and generalize as necessary and in a way supported by existing (or just written) code.
In |
> functions that modify data structures take structures as the last argument (because you commonly have to curry them)
This is true of basic functions in functional programming, but easy currying is not the ultimate merit of a function.
Oh, come on, look at the new patch, 3 out of four uses of `addPassthru` in nixpkgs actually benefit from currying.
Agreement of the name with the order of arguments is more important.
Indeed, which is why I fixed `addPassthru` in the new patch to follow the convention used by all other functions modifying data.
Moreover, often currying is more confusing than useful, especially if the number of arguments is large, such as in your 5 argument version of `extendDerivation`, and it imposes the argument order that hinders reading in the typical non-carried case.
I don't see how "the last argument is always the thing you modify" can be confusing. Its a common convention. `addPassthru` was confusing to me before the fix.
> But I'd rather add a function that covers all the use cases I can come up with, than introduce copy-paste later
You don't have to choose between these two alternatives. It is bad to copy and paste (not always), but it is also bad to support imaginary use cases at the cost of additional complexity of mysterious features that make one question "what is this for? where is it used? oh, it is not?". It is better to abstract and generalize as necessary and in a way supported by existing (or just written) code.
All that "complexity" you're bikeshedding about here is one beta-reduction per argument. I fail to see how this counts as "complexity".
But, whatever, if other people agree with you, I can remove `f d o` with a patch that I will revert as soon as I go back to patching check-meta.
In `customization.nix` only `addPassthru` and `hydraJob` modify outputs, and they seem dissimilar enough to be kept separate.
"Walk outputs and modify them with a function" part looks similar enough to me. But, well, whatever.
|
Btw, while reading this
You don't have to choose between these two alternatives. It is bad to copy and paste (not always), but it is also bad to support imaginary use cases at the cost of additional complexity of mysterious features that make one question "what is this for? where is it used? oh, it is not?". It is better to abstract and generalize as necessary and in a way supported by existing (or just written) code.
I remembered "The kernel report" talk by Jonathan Corbet (https://www.youtube.com/watch?v=oaZdarL9ttM) where when asked something like "Why does kernel changes so much code so frequently?" he answers "Mostly, lack of foresight."
|
I remembered "The kernel report" talk by Jonathan Corbet (https://www.youtube.com/watch?v=oaZdarL9ttM) where when asked something like "Why does kernel changes so much code so frequently?" he answers "Mostly, lack of foresight."
Oops, should've been more careful with grep. That was another speaker and another talk: Greg Kroah Hartman "The Linux Kernel" https://www.youtube.com/watch?v=L2SED6sewRw.
Anyway, both are worth the time, IMO.
|
777bf71
to
bba1d07
Compare
I'd very much like @nbp to review this prior to merging. |
This is not bikeshedding, this is @orivej trying to keep the tree maintainable, which I think is very commendable. I agree with @orivej that function with 5 arguments is at the very least syntactically complex (as in, argument order has meaning), and this is undesirable. I think that if there is a need for more than 3 arguments, generally attrset should be used instead. I also agree on the general note that abstractions should be introduced on as-needed basis. There is a middle ground between too few abstractions and an ivory tower of abstractions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have been bitten by this issue many times in the past, mostly due to other platform packages.
One question still, is there any use of having an eager check, or this only matters for packages which are in the dependency tree. For example, one use-case of an eager validity check might be to completely refuse the addition of packages having a specific license in Nixpkgs code.
in commonAttrs // { | ||
outputUnspecified = true; | ||
drvPath = assert condition; drv.drvPath; | ||
outPath = assert condition; drv.outPath; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow-up: I think exposing the condition
here might be helpful in the future as a way to filter out bad packages from the input lists, such as preventing X libraries to be used.
Note, In the case of check-meta.nix
, this assert condition;
statement is only used to enforce the strictness of the condition, as it will throw before the assertion can catch it. There is no proper false
value returned, unless the config.handleEvalIssue
function is set to return it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check-meta.nix
will throw before the assertion can catch it
check-meta.nix
is being changed in this PR to return a boolean.
pkgs/stdenv/generic/check-meta.nix
Outdated
|
||
let | ||
attrs = mkDerivationArg; # TODO: probably get rid of passing this one | ||
attrs = derivationArg // { inherit meta; }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Revert this line, or keep it for another patch. Doing this will actually create a new attribute set based on derivationArg
only to lookup the meta
attribute, while doing this on the mkDerivationArg
will not create any new attribute set.
I have been bitten by this issue many times in the past, mostly due to other platform packages.
One question still, is there any use of having an eager check, or this only matters for packages which are in the dependency tree. For example, one use-case of an eager validity check might be to completely refuse the addition of packages having a specific license in Nixpkgs code.
It still would completely refuse to evaluate them even after this patchset, but it will allow peeking at their attrs. I'm unsure why would you want to completely refuse any access to any part of the derivation. Unless, of course, you don't want to be able to even `fetchurl` the offending sources. In which case I would recommend moving `meta.license` into the `fetchurl` and patch `mkDerivation` to inherit `meta` from its `src` by default (will help with automatically generated `homepage`s for `fetchFromGitHub` too, btw).
outputsList = map outputToAttrListElement outputs;
- in commonAttrs // { outputUnspecified = true; };
+ in commonAttrs // {
+ outputUnspecified = true;
+ drvPath = assert condition; drv.drvPath;
+ outPath = assert condition; drv.outPath;
follow-up: I think exposing the `condition` here might be helpful in the future
You mean, expose a separate `condition` check for the top-level derivation? Please elaborate.
as a way to filter out bad packages from the input lists, such as preventing X libraries to be used.
I do want this feature too, but I'm yet unsure on what is the best way to proceed here. My first though was to use `meta.tags` or similar for this and to allow banning packages by blacklisting them. E.g. I would love to tag everything made by Poettering with a "sucks" tag and blacklist it.
But there's a more uniform way: allow arbitrary filtering functions, e.g. (won't work, but you get the idea)
```
nixpkgs.config.filter = p: elem p.meta.license licenseWhitelist;
```
and move all the white-listing and black-listing out of `check-meta.nix` into the lib for people to make their own filters.
Anyway, this can be discussed after we merge this patchset, I really want my `relatedPackages` patchset in master ASAP.
Note, In the case of `check-meta.nix`, this `assert condition;` statement is only used to enforce the strictness of the condition, as it will throw before the assertion can catch it. There is no proper `false` value returned, unless the `config.handleEvalIssue` function is set to return it.
Correct. It is by design. If `check-meta.nix` or `handleEvalIssue` throws with a descriptive message => that's good, if it simply fails => one would get an assertion error at the very least.
let
- attrs = mkDerivationArg; # TODO: probably get rid of passing this one
+ attrs = derivationArg // { inherit meta; };
Revert this line, or keep it for another patch. Doing this will actually create a new attribute set based on `derivationArg` only to lookup the `meta` attribute, while doing this on the `mkDerivationArg` will not create any new attribute set.
Ah, I didn't think about that. Reverted for now.
|
bba1d07
to
20028b7
Compare
- [X] The implementation is made simpler.
- [X] Order of arguments is fixed.
- [X] mkDerivationArg part reverted.
- [X] Checks pass.
What's holding this now?
|
Unfortunately this cripples filtering in This issue of evaluation checks got really more complicated than I anticipated and I didn't manage to fix all problems before getting distracted by more urgent problems, though I do plan to get back to it. BTW, changing |
Unfortunately this cripples filtering in `nix-env -qa`
That's unfortunate, indeed.
TL; DR: my conclusion there was to put the checks on the point when the `name` attribute is evaluated, as that's one of the few that `nix-env` does touch.
Unfortunately, #32424 will still be broken by forcing checks on `name` access.
BTW, changing `nix-env` would be cleaner
I see no other way then, to be honest.
|
changing `nix-env` would be cleaner, but such a change would be probably much more difficult to push through, especially to 1.11.*
A question after reading #22277.
Why can't we just add `pkg.strict` attr with `extendDerivation` (or `mkDerivation`) that is equivalent to the current top-level strict attr access on master? This way `nix-env` will need a tiny backwards-compatible change of `(pkg.strict or pkg).name` instead of `pkg.name`.
Old `nix-env`s will list broken packages on new `nixpkgs`, but who cares.
A much more ugly way is to put all the effects of this patchset into `pkg.lazy` attr and use that for `relatedPackages`. No changes to `nix-env` are going to be needed, but the rest is going to be really ugly (lazification as here first, followed by copy-paste into `.lazy`, followed by forcing some of the attrs in the top-level drv just for `nix-env`, ugh).
|
The laziness will be useful elsewhere as well – for example ATM you can't even inspect |
The laziness will be useful elsewhere as well – for example ATM you can't even inspect `meta` without running the platform/brokenness/... checks. Over long term, it might be best to have some interface to nix (maybe like the suggested `pkg.strict`), assuming we still want a well-behaved function listing all "available" packages.
So, what is your desired plan of action for this? Should I just add `.strict` into this patchset and then make a PR for `nix` with a patch to `nix-env` or what?
|
@vcunat I started implementing the thing from my previous post, but it turns out that
Unfortunately this cripples filtering in `nix-env -qa`, because `nix-env` touches neither of the two attributes by default.
is incorrect for the current `nix-env`:
```
$ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-env -f ./default.nix -qas --description --argstr system x86_64-linux | grep xen
--- hunspell-dict-eu-es-xuxen-5-2015.11.10 Basque (Xuxen 5)
--- qemu-xen-host-cpu-only-2.11.0 A generic and open source machine emulator and virtualizer
--- qemu-xen-host-cpu-only-2.11.0 A generic and open source machine emulator and virtualizer
--- qemu-xen-host-cpu-only-2.11.0 A generic and open source machine emulator and virtualizer
--- qemu-xen-host-cpu-only-2.11.0 A generic and open source machine emulator and virtualizer
--- xen-4.5.5 Xen hypervisor and related components (vanilla)
--- xen-4.5.5 Xen hypervisor and related components (light)
--- xen-4.5.5 Xen hypervisor and related components (slim)
--- xen-4.8.2 Xen hypervisor and related components (light)
--- xen-4.8.2 Xen hypervisor and related components (slim)
--- xen-4.8.2 Xen hypervisor and related components (vanilla)
$ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-env -f ./default.nix -qas --description --argstr system aarch64-linux | grep xen
--- hunspell-dict-eu-es-xuxen-5-2015.11.10 Basque (Xuxen 5)
```
This patchset is perfect. Lets merge.
|
Nitpick: I suggest that we remove `addPassthru`, as we flipped it, it seems unlikely to be used outside nixpkgs (two call sites there), and `extendDerivation true` is easy enough. (In any case, we should mention it in 18.03 release notes.) What do you think?
I'm ok either way, but, personally, I would deprecate it first and remove on 18.03 (trivial enough to be removed soon, but its rather annoying when evaluation suddenly breaks on update).
To continue from 4166183#commitcomment-26846394
`checkMetaRecursively = true` by default was dismissed because of `nix-env -qa` performance –
that's an intrinsic problem that can't really be fixed even in nix itself (at least not easily; it's
just lots of work to check everything). However, I *do* want it for the checks that @GrahamcOfBorg
is running on every PR (soon-ish).
While its true that 2x slowdown might be too slow for `nix-env -qa` where its has to check all the packages (though, I'm just biting the bullet here, since I usually want more precise results), I think it is totally ok to be recursive by default for the manual build where its checks only a few (< 10 right now in SLNOS, maybe couple hundred eventually).
Since `nix` without NixOS/nix#1771 doesn't look at `meta.evaluates` everything with this PR looks ok to me in this regard.
But I agree that `checkMetaRecursively = true;` by default is not the best idea when combined with NixOS/nix#1771.
Well, another, less stateful, solution to this is to remove `checkMetaRecursively` and to add `meta.evaluatesRecursively` or something similar. This way `checkMetaRecursively` can be made into an option for manual's builder and NixOS/nix#1771 can simply provide another flag to change it per-evaluation.
> Moreover, I think it still makes sense to have both 0720ca5374bebb68048f6cc2532b06a5638e88e0 and bf15abecd64ed553fd865f26cd0bb20e93a8429e even in the presence of `tryEval` because `tryEval` would be expensive. I would just do `checkMetaRecursively = true` by default and fix all cases like
>
> > For example, if one uses `preConfigure = ''foo bar ${package} baz''`.
>
> by adding those `${package}`s to `nativeBuildInputs`. Maybe I would go even further and call all such cases "a bug" and automate to check for cases where `tryEval` and `meta.evaluates` disagree.
I wouldn't expect `tryEval foo` to be noticeably more expensive than `assert foo`, but I haven't tried.
Ah, this kind of `tryEval`. What would be the `foo` for this use case then? If you want precise results what could you evaluate except `tryEval package.outPath`?
In the quoted piece of mine I proposed to just check that `tryEval package.outPath` evaluates iff `meta.evaluatesRecursively` and mark all the places where it does not to be bugs.
|
As you write, it depends on the intentions. |
694d131
to
41df994
Compare
I'm inclined to merge as it is, perhaps after agreeing on the `addPassthru` nitpick. I'd leave `checkMetaRecursively` as it is for now; (maybe with a note that) we can improve its precision later.
`addPassthru`: OK, let me add a warning for now and remove just before 18.03 branch-off.
OK. OK. LGTM.
Updated the branch. Feel free to merge as is, or do something different.
|
Closes #22277 - it's superseded; I have some WIP on evaluation performance, but best do that in a separate PR/thread.
Yay! Cheers! Thanks!
|
🎉 Thank you, too. |
addPassthru became unused in NixOS#33057, but its signature was changed at the same time. This commit restores the original signature and updates the warning and the changelog.
FWIW this (50148f0 in particular) seems to introduced some unfortunate Thoughts on how that might be remedied from either nixpkgs or Nix? Thanks! |
Eww, I guess |
Ugh. I suggest we fix nix this time around, half of reason behind NixOS/nix#1771 didn't go anywhere.
|
Should I open a new issue for this? |
NixOS/nixpkgs#33057 delays `stdenv.mkDerivation` derivation checks (e.g. meta, brokennes, etc) to `.drvPath` and `.outPath` evaluation leaving other fields accessible. That has a side-effect of making `nix-env -qa` list broken packages. This patch allows to - evaluate all `.outPath`s in `-qa` output strictly (`--strict` option) at the cost of 5x-7x slowdown; - disable filtering completely with `--all` option at the cost of getting some junk in the output and some speedup; - filter based on `.meta.available` (the default, `--approximate` option) at the cost of getting a tiny amounts of junk in the output and 0.96x-1.04x slowdown. It is also possible to get near-perfect results with `--approximate` at the cost of 2x slowdown by setting `config.checkMetaRecursively` in nixpkgs.
NixOS/nixpkgs#33057 delays `stdenv.mkDerivation` derivation checks (e.g. meta, brokennes, etc) to `.drvPath` and `.outPath` evaluation leaving other fields accessible. That has a side-effect of making `nix-env -qa` list broken packages. This patch allows to - evaluate all `.outPath`s in `-qa` output strictly (`--strict` option) at the cost of 5x-7x slowdown; - disable filtering completely with `--all` option at the cost of getting some junk in the output and some speedup; - filter based on `.meta.available` (the default, `--approximate` option) at the cost of getting a tiny amounts of junk in the output and 0.96x-1.04x slowdown. It is also possible to get near-perfect results with `--approximate` at the cost of 2x slowdown by setting `config.checkMetaRecursively` in nixpkgs.
Motivation for this change
I want to have #32424 reverted by #33006 because of the bug uncovered in the discussion of #33006.
This patchset seems to fix the problem.
/cc @vcunat @grahamc
Things done