Add --include-build-refs to `nix path-info' command #3523

matthewbauer · 2020-04-22T02:11:38Z

This print out all of the store paths referenced in the Nix
expression.

This is useful with tools like cachix, and replace the previous usages
of:

  $ nix-store -qR --include-outputs $(nix-instantiate '<nixpkgs>' -A hello)

which now become:

  $ nix path-info --recursive --include-build-refs nixpkgs.hello

which can be piped into cachix like:

  $ nix path-info --include-build-refs nixpkgs.hello | cachix push my-cache

It provides a flag for --include-build-refs, meant to parallel dependencies needed for nix build.

Related to cachix/cachix#52
and #3506

/cc @domenkozar @edolstra

This allows use you to get all drvs that your Nix expression references. Output looks like this: $ nix-instantiate --include-ifd warning: you did not specify '--add-root'; the result might be removed by the garbage collector /nix/store/xvplw4i56ys3j1lk4bxnr8r4725rb6c8-bauer-1.5.0.drv /nix/store/nk7j41ldlq39lmmsg5v1mlwwwpnqnbgk-README.drv /nix/store/pp84r1z7j7aqqarbwkq55h5asy35rs15-package-list.drv This can be piped put into nix-store to get all outputs needed: $ nix-store -qR --include-outputs $(nix-instantiate --include-ifd) /nix/store/00crwk6z71l36av0gspzi0z410h8fq1r-libXtst-1.2.3.tar.bz2 /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh /nix/store/58y89v7rl254dc2cygcfd5wzhv0kjm4m-bash44-013.drv /nix/store/7c0yirypq720qgj2clyanqp3b18h1lj0-bison-3.4.2.tar.gz.drv /nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv /nix/store/c0sr4qdy8halrdrh5dpm7hj05c6hyssa-unpack-bootstrap-tools.sh /nix/store/drsdq2ca1q1dj1hd0r1w2hl4s0fak1vh-bootstrap-tools.tar.xz.drv /nix/store/bfil786fxmnjcwc7mqpm0mk4xnm2cphg-bootstrap-tools.drv ...

This print out all of the store paths referenced in the Nix expression. Examples provided in --help. This is useful with tools like cachix, and replace the previous usages of: $ nix-store -qR --include-outputs $(nix-instantiate '<nixpkgs>' -A hello) which now become: $ nix refs --build --run nixpkgs.hello which can be piped into cachix like: $ nix refs --build --run nixpkgs.hello | cachix push my-cache Related to cachix/cachix#52 and NixOS#3506

nix-instantiate is deprecated!

edolstra · 2020-04-22T15:29:21Z

Maybe these could be flags to nix path-info? We already have nix path-info -r to show the runtime dependencies.

matthewbauer · 2020-04-22T15:56:05Z

Maybe these could be flags to nix path-info? We already have nix path-info -r to show the runtime dependencies.

That might make sense, but right now nix path-info wants store paths. To calculate, build and eval-time dependencies, we need an expression.

For build-time dependencies, we actually could use the "deriver" field to get that derivation, but it's frequently missing. Also eval-time will always need an expression (unless we add something like a "deriver-expression" field)

matthewbauer · 2020-04-22T19:44:10Z

Perhaps we could rename it to nix expr-info?

gleber · 2020-04-22T19:50:15Z

Maybe call it `nix info` and make it take both expressions and nix store paths?

…

On Wed, 22 Apr 2020 at 21:44, Matthew Bauer ***@***.***> wrote: Perhaps we could rename it to nix expr-info? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3523 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIDIO5LEEWKIMAEEVBNF3RN5CJTANCNFSM4MNYQ6VA> .

edolstra · 2020-04-23T09:03:58Z

nix path-info (like most nix commands) works on a lot more than store paths:

$ nix path-info nixpkgs#hello
/nix/store/y26qxcq1gg2hrqpxdc58b2fghv2bhxjg-hello-2.10

$ nix path-info -f ~/Dev/nixpkgs hello
/nix/store/7hz2ms6mxdbfwj8bl4fcd171lly4l9wx-hello-2.10

$ nix path-info --impure --expr '(import <nixpkgs> {}).hello'
/nix/store/rr3y0c6zyk7kjjl8y19s4lsrhn4aiq1z-hello-2.10

Both require --recursive. --build calculates all buildtime dependencies of a path. --eval calculates all evaltime dependencies of an expression.

matthewbauer · 2020-04-23T16:24:15Z

Updated to use nix path-info, instead of adding a new command.

src/nix/command.cc

Now used in nix/command.cc error

matthewbauer · 2020-05-18T21:03:11Z

@edolstra Let me know if any concerns remain. It should be ready for re-review.

If we still have the .drv, it will be available in our database under deriver. We can use that instead of requiring we actually received a drvPath.

matthewbauer · 2020-06-11T17:42:25Z

Updated to support --build with plain output paths. Also refactored data types a bit.

src/libexpr/primops.cc

src/nix/command.cc

Co-authored-by: John Ericson <git@JohnEricson.me>

Ericson2314 · 2020-06-11T22:32:28Z

Hmm i actually think --eval alone should get all imported paths, so e.g. import (builtins.fetchurl ...) doesn't slip under the radar. The combination of --eval and --build can turn those paths into derivations where possible.

Even non-derivations are important to have in this output, so keep them in our list.

This is more accurate since things like fileExists and readFile also go through this.

Runtime deps aren’t always useful, especially in the case of creating gc roots for eval deps. for instance: $ nix-store --add-result --indirect -r $(nix path-info --recursive --no-run --eval my-expression)

Ericson2314 · 2020-06-12T00:13:09Z

src/libexpr/primops.cc

@@ -54,6 +54,7 @@ void EvalState::realiseContext(const PathSet & context)
        auto ctx = store->parseStorePath(decoded.first);
        if (!store->isValidPath(ctx))
            throw InvalidPathError(store->printStorePath(ctx));
+        importedPaths.push_back(ctx.clone());


I think we would want to push the path to the specific output rather than drv file in this case? for --build --eval would we get the original drv file again.

It may be useful to have more information, but this is all that's needed for the computeFSClosure. The original drv needs to be included to get dependencies for --build. It may end up getting in multiple times, but I think that's okay.

From counting the number included, I am fairly sure it is the right thing. For instance, here's the deps I get for each:

recursive evaltime buildtime runtime total

Y N N N 0

Y Y N N 218

Y N N Y 293

Y Y N N 511

Y N Y X 1003

Y Y Y X 1008

What standard commands or flags would enumerate all of these, but it can be unclear how to get a report/listing of each row. How was this generated?

src/libexpr/primops.cc

matthewbauer · 2020-06-12T03:30:35Z

Does anyone have objections to the naming of the options here? --eval, --build, and --no-run make sense to me but I realized it is a little confusing because you're not actually doing those things - just recursing into them. Perhaps --recursive-eval, --recursive-build are better... I also thought about something like --closure-type=runtime/build/eval.

edolstra · 2020-06-12T09:33:46Z

I'm hesitant about adding these flags:

Build dependencies expose the concept of store derivations, which the nix command tries to move away from (so that in the future, we could stop materializing them to disk, and instead just send an in-memory graph of build actions to the daemon).
Evaluation-time dependencies are unpredictable: if evaluation caching is enabled (the default), then nix path-info could output different results when you run it a second time.

Ericson2314 · 2020-06-12T13:10:31Z

Sorry I must be missing something. Build-time dependencies still exists as a concept, whether or not we have the drv files. Is the issue using drv file paths to refer to derivations? Or just that the daemon might not remember how something was built?
Could we put this info in the evaluation cache so it can be retrieved later? At the end of the day, we (and I assume others) will be happy with any approve that allows one to be sure they are saving enough dependencies to make importing derivations reproducible----this is a feature we really want. If there is a different approach you would prefer, we'd be happy to try that instead.

This could have some bad interactions with the eval cache. Once flakes is merged, we can resolve those so that --eval will always reevaluate the expression.

matthewbauer · 2020-06-26T17:31:11Z

Updated to just provide --build since --eval could have some bad interactions with the eval cache.

domenkozar · 2020-10-09T14:09:47Z

@edolstra there's no way to do this currently with flakes, which really limits ways to inspect closures for build dependencies.

If we stop materializing to disk, we can still compute the build closure it just can't be cached?

hyperfekt · 2020-10-09T21:04:31Z

Can --eval not be made to run without the evaluation cache? I'm thinking it's still going to be very useful.

matthewbauer · 2021-01-29T23:36:46Z

@domenkozar @edolstra Any chance this can get some more review? I've updated on to master and now renamed it to --include-build-refs. I'm hoping the longer, more specific name means it's a little less objectionable and more obvious how it fits into the --recursive flag flow.

Continuation of NixOS#3523, but with evaluation-time refs this time. More discussion is available at that PR. This keeps track of imported paths when evaluating. When --recursive & --include-eval-refs is passed, these paths & their references are included in the resulting closure.

domenkozar · 2021-01-30T10:04:12Z

@matthewbauer does it address concerns raised by @edolstra? I'm +1 for merging it otherwise.

This primop lets you get information on paths realised (pathExists, import, readFile) by Nix. It tracks paths realised in Nix & puts the results under the "paths" attr. For instance: $ nix eval --impure --expr '(builtins.traceContext (import ./.).outPath)' { paths = [ "/nix/store/2mpgi4bvn8py4liv9w3mjxd2c5r7bvv8-source" "/nix/store/3k7i1rdqcgyzjxvqm37hapdidy4ls4s3-source" "/nix/store/kqxic0j6wpsaw2bb51hr1yc1nb1z2xw8-source" ]; value = "/nix/store/3k7i1rdqcgyzjxvqm37hapdidy4ls4s3-source"; } This is an alternative to the --include-eval-refs from NixOS#3523 & --include-ifd from NixOS#3506.

matthewbauer added 3 commits April 16, 2020 15:28

Remove --include-ifd from nix-instantiate

2eb9e07

nix-instantiate is deprecated!

matthewbauer mentioned this pull request Apr 22, 2020

Add --include-ifd option to nix-instantiate #3506

Closed

Use toBuildables to ensure we have derivation

abe2d7f

matthewbauer added 2 commits April 23, 2020 11:06

Remove ‘nix refs’

18182a7

Add --build and --eval flag to “nix path-info”

4eeddcd

Both require --recursive. --build calculates all buildtime dependencies of a path. --eval calculates all evaltime dependencies of an expression.

matthewbauer changed the title ~~Add `nix refs' command~~ Add --build and --eval to `nix path-info' command Apr 23, 2020

Merge remote-tracking branch 'origin/master' into nix-refs

f25f023

matthewbauer commented Apr 23, 2020

View reviewed changes

src/nix/command.cc Outdated Show resolved Hide resolved

domenkozar reviewed Apr 24, 2020

View reviewed changes

src/nix/command.cc Outdated Show resolved Hide resolved

src/nix/command.cc Outdated Show resolved Hide resolved

src/nix/command.cc Outdated Show resolved Hide resolved

Add "what()" for Buildable

93684d5

Now used in nix/command.cc error

domenkozar requested a review from edolstra April 30, 2020 17:11

domenkozar added improvement new-cli Relating to the "nix" command labels Apr 30, 2020

matthewbauer added 2 commits June 11, 2020 12:23

Use proper data type for importedDrvs

3a155a9

Support getting build paths from output path

98d5a3b

If we still have the .drv, it will be available in our database under deriver. We can use that instead of requiring we actually received a drvPath.

Ericson2314 reviewed Jun 11, 2020

View reviewed changes

src/libexpr/primops.cc Outdated Show resolved Hide resolved

Ericson2314 reviewed Jun 11, 2020

View reviewed changes

src/nix/command.cc Outdated Show resolved Hide resolved

Ericson2314 mentioned this pull request Jun 11, 2020

Extend Rust FFI #3450

Closed

Update src/nix/command.cc

34fc432

Co-authored-by: John Ericson <git@JohnEricson.me>

matthewbauer added 4 commits June 11, 2020 18:26

Include all paths in --eval

240b66a

Even non-derivations are important to have in this output, so keep them in our list.

Rename importedPaths to realisedPaths

4e2e3fa

This is more accurate since things like fileExists and readFile also go through this.

Add option to path-info to skip run deps

d52e556

Runtime deps aren’t always useful, especially in the case of creating gc roots for eval deps. for instance: $ nix-store --add-result --indirect -r $(nix path-info --recursive --no-run --eval my-expression)

Only include derivers for --build

bf02207

Ericson2314 reviewed Jun 12, 2020

View reviewed changes

src/libexpr/primops.cc Outdated Show resolved Hide resolved

Add error when --no-run and --build are combined

c2f0229

Ericson2314 mentioned this pull request Jun 12, 2020

Add a way to get all the outputs of a derivation with their label #3678

Merged

matthewbauer added 2 commits June 26, 2020 13:07

Merge remote-tracking branch 'origin/master' into nix-refs

0ddf019

Remove --eval support

6c591a8

This could have some bad interactions with the eval cache. Once flakes is merged, we can resolve those so that --eval will always reevaluate the expression.

matthewbauer changed the title ~~Add --build and --eval to `nix path-info' command~~ Add --build to `nix path-info' command Jun 26, 2020

Merge branch 'master' into nix-refs

7582950

matthewbauer added 2 commits January 29, 2021 17:32

Merge remote-tracking branch 'origin/master' into nix-refs

141dfb0

Rename --build to --include-build-refs

1012ab2

matthewbauer changed the title ~~Add --build to `nix path-info' command~~ Add --include-build-refs to `nix path-info' command Jan 29, 2021

Try to fix build error

86b65fc

matthewbauer force-pushed the nix-refs branch from fcf4fe6 to 86b65fc Compare January 30, 2021 02:38

matthewbauer mentioned this pull request Jan 30, 2021

Add --include-eval-refs to StorePathCommands #4496

Open

matthewbauer mentioned this pull request Apr 21, 2021

Add traceContext primop #4728

Open

matthewbauer closed this Apr 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --include-build-refs to `nix path-info' command #3523

Add --include-build-refs to `nix path-info' command #3523

matthewbauer commented Apr 22, 2020 •

edited

edolstra commented Apr 22, 2020

matthewbauer commented Apr 22, 2020

matthewbauer commented Apr 22, 2020

gleber commented Apr 22, 2020 via email

edolstra commented Apr 23, 2020

matthewbauer commented Apr 23, 2020

matthewbauer commented May 18, 2020

matthewbauer commented Jun 11, 2020

Ericson2314 commented Jun 11, 2020

Ericson2314 Jun 12, 2020

matthewbauer Jun 12, 2020

tomberek Dec 28, 2021

matthewbauer commented Jun 12, 2020 •

edited

edolstra commented Jun 12, 2020

Ericson2314 commented Jun 12, 2020

matthewbauer commented Jun 26, 2020

domenkozar commented Oct 9, 2020

hyperfekt commented Oct 9, 2020

matthewbauer commented Jan 29, 2021

domenkozar commented Jan 30, 2021

Add --include-build-refs to `nix path-info' command #3523

Add --include-build-refs to `nix path-info' command #3523

Conversation

matthewbauer commented Apr 22, 2020 • edited

edolstra commented Apr 22, 2020

matthewbauer commented Apr 22, 2020

matthewbauer commented Apr 22, 2020

gleber commented Apr 22, 2020 via email

edolstra commented Apr 23, 2020

matthewbauer commented Apr 23, 2020

matthewbauer commented May 18, 2020

matthewbauer commented Jun 11, 2020

Ericson2314 commented Jun 11, 2020

Ericson2314 Jun 12, 2020

Choose a reason for hiding this comment

matthewbauer Jun 12, 2020

Choose a reason for hiding this comment

tomberek Dec 28, 2021

Choose a reason for hiding this comment

matthewbauer commented Jun 12, 2020 • edited

edolstra commented Jun 12, 2020

Ericson2314 commented Jun 12, 2020

matthewbauer commented Jun 26, 2020

domenkozar commented Oct 9, 2020

hyperfekt commented Oct 9, 2020

matthewbauer commented Jan 29, 2021

domenkozar commented Jan 30, 2021

matthewbauer commented Apr 22, 2020 •

edited

matthewbauer commented Jun 12, 2020 •

edited