Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdenv: perform checks only when evaluating .drv and .out #33057

Merged
merged 10 commits into from Jan 14, 2018

Conversation

oxij
Copy link
Member

@oxij oxij commented Dec 25, 2017

Motivation for this change

I want to have #32424 reverted by #33006 because of the bug uncovered in the discussion of #33006.
This patchset seems to fix the problem.

/cc @vcunat @grahamc

Things done
  • Both pass and fail checks evaluate fine.
  • No hashes change.
  • Fits CONTRIBUTING.md.

Copy link
Contributor

@orivej orivej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The effect of this PR is good, but the implementation seems a bit too complex.

addPassthru = drv: passthru:
the derivation itself and then map all outputs with given
functions. */
extendDerivation = passthru: f: d: o: drv:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the name of the function, drv should be the first parameter.

None of the f, d, o use the first of the two arguments this function passes to them, so it should not pass it at all. They also simply return their second argument (what else can be done with an output path?), so it should not be passed either. The new function needs just one new argument isValid (or { isValid ? true; }) to use as:

drvPath = assert isValid; drv.${outputName}.drvPath;
outPath = assert isValid; drv.${outputName}.outPath;

@oxij
Copy link
Member Author

oxij commented Dec 25, 2017 via email

@oxij
Copy link
Member Author

oxij commented Dec 25, 2017 via email

@orivej
Copy link
Contributor

orivej commented Dec 25, 2017

functions that modify data structures take structures as the last argument (because you commonly have to curry them)

This is true of basic functions in functional programming, but easy currying is not the ultimate merit of a function. Agreement of the name with the order of arguments is more important. Moreover, often currying is more confusing than useful, especially if the number of arguments is large, such as in your 5 argument version of extendDerivation, and it imposes the argument order that hinders reading in the typical non-carried case.

But I'd rather add a function that covers all the use cases I can come up with, than introduce copy-paste later

You don't have to choose between these two alternatives. It is bad to copy and paste (not always), but it is also bad to support imaginary use cases at the cost of additional complexity of mysterious features that make one question "what is this for? where is it used? oh, it is not?". It is better to abstract and generalize as necessary and in a way supported by existing (or just written) code.

customization.nix has copious amounts of copy-paste already, just look at the code there, like all the functions that modify outputs follow the same pattern, I would just generalize-rewrite half the functions there if I weren't sure it would cause the usual amounts of bikeshedding

In customization.nix only addPassthru and hydraJob modify outputs, and they seem dissimilar enough to be kept separate. If your generalization would increase both the line count and the effort to understand concrete uses, it will have negative value even if it decreased copy-paste.

@oxij
Copy link
Member Author

oxij commented Dec 26, 2017 via email

@oxij
Copy link
Member Author

oxij commented Dec 26, 2017 via email

@oxij
Copy link
Member Author

oxij commented Dec 26, 2017 via email

@oxij oxij force-pushed the stdenv/delay-check-meta branch 3 times, most recently from 777bf71 to bba1d07 Compare December 26, 2017 11:12
@grahamc
Copy link
Member

grahamc commented Dec 26, 2017

I'd very much like @nbp to review this prior to merging.

@lukateras
Copy link
Member

lukateras commented Dec 26, 2017

This is not bikeshedding, this is @orivej trying to keep the tree maintainable, which I think is very commendable.

I agree with @orivej that function with 5 arguments is at the very least syntactically complex (as in, argument order has meaning), and this is undesirable. I think that if there is a need for more than 3 arguments, generally attrset should be used instead.

I also agree on the general note that abstractions should be introduced on as-needed basis. There is a middle ground between too few abstractions and an ivory tower of abstractions.

Copy link
Member

@nbp nbp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been bitten by this issue many times in the past, mostly due to other platform packages.

One question still, is there any use of having an eager check, or this only matters for packages which are in the dependency tree. For example, one use-case of an eager validity check might be to completely refuse the addition of packages having a specific license in Nixpkgs code.

in commonAttrs // {
outputUnspecified = true;
drvPath = assert condition; drv.drvPath;
outPath = assert condition; drv.outPath;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow-up: I think exposing the condition here might be helpful in the future as a way to filter out bad packages from the input lists, such as preventing X libraries to be used.

Note, In the case of check-meta.nix, this assert condition; statement is only used to enforce the strictness of the condition, as it will throw before the assertion can catch it. There is no proper false value returned, unless the config.handleEvalIssue function is set to return it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check-meta.nix will throw before the assertion can catch it

check-meta.nix is being changed in this PR to return a boolean.


let
attrs = mkDerivationArg; # TODO: probably get rid of passing this one
attrs = derivationArg // { inherit meta; };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this line, or keep it for another patch. Doing this will actually create a new attribute set based on derivationArg only to lookup the meta attribute, while doing this on the mkDerivationArg will not create any new attribute set.

@oxij
Copy link
Member Author

oxij commented Dec 27, 2017 via email

@oxij
Copy link
Member Author

oxij commented Dec 28, 2017 via email

@vcunat
Copy link
Member

vcunat commented Dec 30, 2017

Unfortunately this cripples filtering in nix-env -qa, because nix-env touches neither of the two attributes by default. I've fought that in #22277. TL; DR: my conclusion there was to put the checks on the point when the name attribute is evaluated, as that's one of the few that nix-env does touch. EDIT: the effect on the new nix search command seems the same, by a quick test.

This issue of evaluation checks got really more complicated than I anticipated and I didn't manage to fix all problems before getting distracted by more urgent problems, though I do plan to get back to it. BTW, changing nix-env would be cleaner, but such a change would be probably much more difficult to push through, especially to 1.11.

@oxij
Copy link
Member Author

oxij commented Dec 30, 2017 via email

@oxij
Copy link
Member Author

oxij commented Dec 30, 2017 via email

@vcunat
Copy link
Member

vcunat commented Dec 30, 2017

The laziness will be useful elsewhere as well – for example ATM you can't even inspect meta without running the platform/brokenness/... checks. Over long term, it might be best to have some interface to nix (maybe like the suggested pkg.strict), assuming we still want a well-behaved function listing all "available" packages.

@oxij
Copy link
Member Author

oxij commented Dec 30, 2017 via email

@oxij
Copy link
Member Author

oxij commented Dec 31, 2017 via email

@oxij
Copy link
Member Author

oxij commented Jan 14, 2018 via email

@vcunat
Copy link
Member

vcunat commented Jan 14, 2018

addPassthru: OK, let me add a warning for now and remove just before 18.03 branch-off.

Ah, this kind of tryEval. What would be the foo for this use case then?

As you write, it depends on the intentions. tryEval is just a way to catch the exceptions and covert them to booleans.

@oxij
Copy link
Member Author

oxij commented Jan 14, 2018 via email

@vcunat vcunat merged commit 41df994 into NixOS:master Jan 14, 2018
vcunat added a commit that referenced this pull request Jan 14, 2018
Closes #22277 - it's superseded;  I have some WIP on evaluation
performance, but best do that in a separate PR/thread.
@oxij
Copy link
Member Author

oxij commented Jan 14, 2018 via email

@vcunat
Copy link
Member

vcunat commented Jan 14, 2018

🎉 Thank you, too.

orivej added a commit to orivej/nixpkgs that referenced this pull request Jan 24, 2018
addPassthru became unused in NixOS#33057, but its signature was changed at the same
time.  This commit restores the original signature and updates the warning and
the changelog.
@orivej orivej mentioned this pull request Jan 24, 2018
8 tasks
@dtzWill
Copy link
Member

dtzWill commented Mar 11, 2018

FWIW this (50148f0 in particular) seems to introduced some unfortunate nix-env behavior: NixOS/nix#1963

Thoughts on how that might be remedied from either nixpkgs or Nix? Thanks!

@vcunat
Copy link
Member

vcunat commented Mar 12, 2018

Eww, I guess nix-env silently skips assertions but not throws? It's possible I only tested this with repl and build.

@oxij
Copy link
Member Author

oxij commented Mar 12, 2018 via email

@dtzWill
Copy link
Member

dtzWill commented Mar 15, 2018

Should I open a new issue for this?

oxij added a commit to oxij/nix that referenced this pull request Jun 12, 2018
NixOS/nixpkgs#33057 delays `stdenv.mkDerivation` derivation checks (e.g.
meta, brokennes, etc) to `.drvPath` and `.outPath` evaluation leaving
other fields accessible. That has a side-effect of making `nix-env -qa`
list broken packages.

This patch allows to

- evaluate all `.outPath`s in `-qa` output strictly (`--strict` option)
  at the cost of 5x-7x slowdown;
- disable filtering completely with `--all` option at the cost of getting some
  junk in the output and some speedup;
- filter based on `.meta.available` (the default, `--approximate` option)
  at the cost of getting a tiny amounts of junk in the output and
  0.96x-1.04x slowdown.

It is also possible to get near-perfect results with `--approximate` at
the cost of 2x slowdown by setting `config.checkMetaRecursively` in nixpkgs.
oxij added a commit to oxij/nix that referenced this pull request Jun 12, 2018
NixOS/nixpkgs#33057 delays `stdenv.mkDerivation` derivation checks (e.g.
meta, brokennes, etc) to `.drvPath` and `.outPath` evaluation leaving
other fields accessible. That has a side-effect of making `nix-env -qa`
list broken packages.

This patch allows to

- evaluate all `.outPath`s in `-qa` output strictly (`--strict` option)
  at the cost of 5x-7x slowdown;
- disable filtering completely with `--all` option at the cost of getting some
  junk in the output and some speedup;
- filter based on `.meta.available` (the default, `--approximate` option)
  at the cost of getting a tiny amounts of junk in the output and
  0.96x-1.04x slowdown.

It is also possible to get near-perfect results with `--approximate` at
the cost of 2x slowdown by setting `config.checkMetaRecursively` in nixpkgs.
@oxij oxij deleted the stdenv/delay-check-meta branch September 8, 2018 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants