treewide: with stdenv.lib; in meta -> with lib; #108978

Profpatsch · 2021-01-10T21:07:14Z

meta = with stdenv.lib;

is a widely used pattern. We want to slowly remove
the stdenv.lib indirection and encourage people
to use lib directly. Thus let’s start with the meta
field.

This used a rewriting script to mostly automatically
replace all occurances of this pattern, and add the
lib argument to the package header if it doesn’t
exist yet.

The script in its current form is available at
https://cs.tvl.fyi/depot@2f807d7f141068d2d60676a89213eaa5353ca6e0/-/blob/users/Profpatsch/nixpkgs-rewriter/default.nix

Motivation for this change

Things done

Profpatsch · 2021-01-10T21:13:57Z

Strategies used to sanity check the changes involve using a script to randomly evaluate packages, so all important packages should still evaluate.

nix eval --raw '(let pkgs = import ./. {}; in builtins.toJSON (builtins.attrNames pkgs))' \
   | jq -r '.[]' \
   | shuf \
    | head -n 1000 \
     | xargs -n1 nix-instantiate -A

Some leaf packages might have problems if they don’t follow the callPackage standard but pass arguments manually. Hopefully ofborg finds them.

Profpatsch · 2021-01-10T21:18:03Z

The output of

$ git diff --numstat HEAD^ | sort -n | tail -n30
2	2	pkgs/tools/X11/xvkbd/default.nix
2	2	pkgs/tools/X11/xwallpaper/default.nix
2	2	pkgs/tools/X11/xwinwrap/default.nix
3	3	pkgs/applications/audio/clementine/default.nix
3	3	pkgs/applications/misc/ape/default.nix
3	3	pkgs/applications/misc/plover/default.nix
3	3	pkgs/applications/networking/instant-messengers/salut-a-toi/requirements.nix
3	3	pkgs/applications/science/logic/coq/default.nix
3	3	pkgs/applications/video/kodi/plugins.nix
3	3	pkgs/development/tools/parsing/antlr/4.8.nix
3	3	pkgs/games/openxray/default.nix
3	3	pkgs/games/riko4/default.nix
3	3	pkgs/misc/vim-plugins/overrides.nix
3	3	pkgs/os-specific/linux/guvcview/default.nix
3	3	pkgs/servers/kippo/default.nix
3	3	pkgs/tools/package-management/nix-prefetch-scripts/default.nix
3	3	pkgs/tools/typesetting/tex/texlive/bin.nix
4	4	pkgs/applications/audio/dfasma/default.nix
4	4	pkgs/applications/misc/dockbarx/default.nix
4	4	pkgs/applications/misc/subsurface/default.nix
4	4	pkgs/applications/virtualization/qemu/default.nix
4	4	pkgs/misc/tmux-plugins/default.nix
5	5	pkgs/tools/text/gawk/default.nix
7	7	pkgs/applications/networking/znc/modules.nix
8	8	pkgs/applications/video/vdr/plugins.nix
9	9	pkgs/misc/vscode-extensions/default.nix
10	10	pkgs/applications/editors/jetbrains/default.nix
16	16	pkgs/applications/misc/octoprint/plugins.nix
26	26	pkgs/applications/editors/eclipse/plugins.nix

shows that there are only a few files which got more than two lines of changes.

AndersonTorres · 2021-01-10T21:33:13Z

Good to know. Following.

SuperSandro2000 · 2021-01-10T21:39:34Z

EditorCheck fails with Error: Argument list too long. I think it is save to ignore it for this PR.

deliciouslytyped · 2021-01-10T21:43:36Z

Couldn't you test it by generating all the drvs and making sure they're invariant? This should apply generally to refactorings that should not change semantics. I don't know if generating all drvs is hard or something but if yes, https://discourse.nixos.org/t/using-nixos-in-an-isolated-environment/3369 is probably relevant.

Profpatsch · 2021-01-11T07:55:34Z

I did a dudu while rebasing, I think it’s fixed now.

Profpatsch · 2021-01-11T08:35:05Z

Thanks to @cole-h running the ofborg check locally, full evaluation should now work again.

Profpatsch · 2021-01-11T08:51:37Z

Waiting for ofborg eval to be green, then rebasing & merging (otherwise there will be new merge conflicts before ofborg evals)

Part of: NixOS#108938 meta = with stdenv.lib; is a widely used pattern. We want to slowly remove the `stdenv.lib` indirection and encourage people to use `lib` directly. Thus let’s start with the meta field. This used a rewriting script to mostly automatically replace all occurances of this pattern, and add the `lib` argument to the package header if it doesn’t exist yet. The script in its current form is available at https://cs.tvl.fyi/depot@2f807d7f141068d2d60676a89213eaa5353ca6e0/-/blob/users/Profpatsch/nixpkgs-rewriter/default.nix

Profpatsch · 2021-01-11T09:28:57Z

ofborg takes a while to reach this eval in its queue, and then chickens out immediately if there is a merge conflict … we are running the https://github.com/nixos/ofborg#running-meta-checks-locally check on a big machine and will merge once that succeeds.

Profpatsch · 2021-01-11T09:38:09Z

Okay, ran the eval and made sure the check from above

    git diff nixpkgs/master...FETCH_HEAD --word-diff=porcelain --word-diff-regex="[a-zA-Z_][a-zA-Z0-9_'-]*" |
        grep -e '^[+-]' |
        grep -ve '^\(+++\|---\)' |
        sort -u

proudces

+lib
-stdenv

alyssais

Ran the OfBorg checks locally. All good.

sternenseemann · 2021-01-11T12:06:42Z

I think a pass with a secondary tool over files changed by this addressing left over stdenv.lib.versionAtLeast etc. usages could be worthwhile.

FRidh · 2021-01-11T18:35:00Z

This broke eval in pkgs/os-specific/linux/nvidia-x11/persistenced.nix blocking hydra on staging-next. Please test treewide changes with hydra-eval-jobs.

broken by #108978

cole-h · 2021-01-11T19:05:12Z

Please test treewide changes with hydra-eval-jobs.

A good idea -- but how (what is the incantation)?

jonringer · 2021-01-12T00:23:33Z

There's probably a better way, but:

nix-shell -p hydra-unstable --run "hydra-eval-jobs -I nixpkgs=$PWD nixos/release.nix"

cole-h · 2021-01-12T00:29:52Z

And how do we know that there's an issue, given that loads of packages fail to evaluate (e.g. unfree and broken)? Is there some pattern that's easily greppable for? Running hydra-eval-jobs -I . pkgs/top-level/release.nix gave me more than 1.6 million lines of output, after filtering out all the broken and unfree packages.

jonringer · 2021-01-12T00:36:26Z

that's an excellent question :)

FRidh · 2021-01-12T07:37:41Z

And how do we know that there's an issue, given that loads of packages fail to evaluate (e.g. unfree and broken)? Is there some pattern that's easily greppable for? Running hydra-eval-jobs -I . pkgs/top-level/release.nix gave me more than 1.6 million lines of output, after filtering out all the broken and unfree packages.

An actual eval error will kill the evaluation. Note one should test pkgs/top-level/release.nix but also those under nixos/. Probably should write a section on this in the Nixpkgs manual.

cole-h · 2021-01-12T07:41:50Z

I see. So as long as hydra-eval-jobs exits zero, all is fine, and one should test both pkgs/top-level/release.nix and nixos/release.nix. Thanks for the help, I'll keep this in mind for the next time. Since I didn't say so earlier: thanks for catching and fixing the issue.

sternenseemann · 2021-01-12T10:58:46Z

An actual eval error will kill the evaluation. Note one should test pkgs/top-level/release.nix but also those under nixos/. Probably should write a section on this in the Nixpkgs manual.

hydra-eval-jobs exits 0 sometimes even when ofborg still finds evaluation errors.

This fixes an evaluation error indroduced by NixOS#108978

Profpatsch · 2021-01-17T20:30:39Z

This is a best-effort refactor as long as we don’t have well-documented ways of making sure the whole tree still evaluates. We used the ofborg check for that, running it locally.

Profpatsch requested review from alyssais and siraben January 10, 2021 21:07

Profpatsch requested review from adisbladis, edwtjo, FRidh, jonringer, mkg20001, mmahut, peti, RaghavSood, rycee, stigtsp and teto as code owners January 10, 2021 21:07

Profpatsch mentioned this pull request Jan 10, 2021

Deprecate use of stdenv.lib across Nixpkgs #108938

Closed

Profpatsch mentioned this pull request Jan 10, 2021

treewide: simplify pkgs.stdenv.lib -> pkgs.lib #108970

Merged

ofborg bot added 6.topic: cinnamon 6.topic: emacs 6.topic: GNOME GNOME desktop environment and its underlying platform 6.topic: lua 6.topic: nixos 6.topic: pantheon The Pantheon desktop environment 6.topic: printing 6.topic: python 6.topic: ruby 6.topic: steam labels Jan 10, 2021

Profpatsch force-pushed the stdlib-lib-rewrite-meta branch 2 times, most recently from ecea912 to 8d15008 Compare January 11, 2021 07:54

Profpatsch force-pushed the stdlib-lib-rewrite-meta branch from 8d15008 to ed6e00d Compare January 11, 2021 08:34

ofborg bot added the 2.status: merge conflict label Jan 11, 2021

Profpatsch force-pushed the stdlib-lib-rewrite-meta branch from ed6e00d to 1a25769 Compare January 11, 2021 09:26

alyssais approved these changes Jan 11, 2021

View reviewed changes

Profpatsch merged commit 4a7f99d into NixOS:master Jan 11, 2021

FRidh added a commit that referenced this pull request Jan 11, 2021

persistenced: fix eval

e3961ff

broken by #108978

FRidh added a commit that referenced this pull request Jan 11, 2021

persistenced: fix eval

4f0da45

broken by #108978

OPNA2608 mentioned this pull request Jan 14, 2021

megapixels: init at 0.14.0 #98479

Merged

10 tasks

B4dM4n mentioned this pull request Jan 14, 2021

nixpart0: stdenv.lib -> lib #109376

Merged

10 tasks

B4dM4n added a commit to B4dM4n/nixpkgs that referenced this pull request Jan 14, 2021

nixpart0: stdenv.lib -> lib

f213aa3

This fixes an evaluation error indroduced by NixOS#108978

B4dM4n mentioned this pull request Jan 14, 2021

salut_a_toi: fix evaluation error + stdenv.lib -> lib #109379

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

treewide: with stdenv.lib; in meta -> with lib; #108978

treewide: with stdenv.lib; in meta -> with lib; #108978

Profpatsch commented Jan 10, 2021

Profpatsch commented Jan 10, 2021

Profpatsch commented Jan 10, 2021

AndersonTorres commented Jan 10, 2021

SuperSandro2000 commented Jan 10, 2021

deliciouslytyped commented Jan 10, 2021 •

edited

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

alyssais left a comment

sternenseemann commented Jan 11, 2021

FRidh commented Jan 11, 2021 •

edited

cole-h commented Jan 11, 2021 •

edited

jonringer commented Jan 12, 2021

cole-h commented Jan 12, 2021

jonringer commented Jan 12, 2021

FRidh commented Jan 12, 2021

cole-h commented Jan 12, 2021

sternenseemann commented Jan 12, 2021

Profpatsch commented Jan 17, 2021

treewide: with stdenv.lib; in meta -> with lib; #108978

treewide: with stdenv.lib; in meta -> with lib; #108978

Conversation

Profpatsch commented Jan 10, 2021

Motivation for this change

Things done

Profpatsch commented Jan 10, 2021

Profpatsch commented Jan 10, 2021

AndersonTorres commented Jan 10, 2021

SuperSandro2000 commented Jan 10, 2021

deliciouslytyped commented Jan 10, 2021 • edited

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

Profpatsch commented Jan 11, 2021

alyssais left a comment

Choose a reason for hiding this comment

sternenseemann commented Jan 11, 2021

FRidh commented Jan 11, 2021 • edited

cole-h commented Jan 11, 2021 • edited

jonringer commented Jan 12, 2021

cole-h commented Jan 12, 2021

jonringer commented Jan 12, 2021

FRidh commented Jan 12, 2021

cole-h commented Jan 12, 2021

sternenseemann commented Jan 12, 2021

Profpatsch commented Jan 17, 2021

deliciouslytyped commented Jan 10, 2021 •

edited

FRidh commented Jan 11, 2021 •

edited

cole-h commented Jan 11, 2021 •

edited