Add --include-ifd option to nix-instantiate #3506

matthewbauer · 2020-04-16T20:32:12Z

This allows use you to get all drvs that your Nix expression
references. Output looks like this:

  $ nix-instantiate --include-ifd
  warning: you did not specify '--add-root'; the result might be removed by the garbage collector
  /nix/store/xvplw4i56ys3j1lk4bxnr8r4725rb6c8-bauer-1.5.0.drv
  /nix/store/nk7j41ldlq39lmmsg5v1mlwwwpnqnbgk-README.drv
  /nix/store/pp84r1z7j7aqqarbwkq55h5asy35rs15-package-list.drv

This can be put into nix-store to get all paths that will be needed to build the Nix expression:

  $ nix-store -qR --include-outputs $(nix-instantiate --include-ifd) | grep -v \.drv$
  /nix/store/00crwk6z71l36av0gspzi0z410h8fq1r-libXtst-1.2.3.tar.bz2
  /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
  /nix/store/c0sr4qdy8halrdrh5dpm7hj05c6hyssa-unpack-bootstrap-tools.sh
  ...

Comes from discussion in #3494.

/cc @domenkozar @Ericson2314

This allows use you to get all drvs that your Nix expression references. Output looks like this: $ nix-instantiate --include-ifd warning: you did not specify '--add-root'; the result might be removed by the garbage collector /nix/store/xvplw4i56ys3j1lk4bxnr8r4725rb6c8-bauer-1.5.0.drv /nix/store/nk7j41ldlq39lmmsg5v1mlwwwpnqnbgk-README.drv /nix/store/pp84r1z7j7aqqarbwkq55h5asy35rs15-package-list.drv This can be piped put into nix-store to get all outputs needed: $ nix-store -qR --include-outputs $(nix-instantiate --include-ifd) /nix/store/00crwk6z71l36av0gspzi0z410h8fq1r-libXtst-1.2.3.tar.bz2 /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh /nix/store/58y89v7rl254dc2cygcfd5wzhv0kjm4m-bash44-013.drv /nix/store/7c0yirypq720qgj2clyanqp3b18h1lj0-bison-3.4.2.tar.gz.drv /nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv /nix/store/c0sr4qdy8halrdrh5dpm7hj05c6hyssa-unpack-bootstrap-tools.sh /nix/store/drsdq2ca1q1dj1hd0r1w2hl4s0fak1vh-bootstrap-tools.tar.xz.drv /nix/store/bfil786fxmnjcwc7mqpm0mk4xnm2cphg-bootstrap-tools.drv ...

edolstra · 2020-04-16T20:51:17Z

Hm, I don't really want to add flags to the legacy commands (nix-instantiate, etc.). Especially when the output is hard to parse (e.g. if I understand this correctly, the output on stdout doesn't distinguish between the regular nix-instantiate output and the IFD derivations). The nix command is intended to have a --json flag whenever appropriate for non-trivial output.

domenkozar · 2020-04-16T20:54:34Z

the output on stdout doesn't distinguish between the regular nix-instantiate output and the IFD derivations

I'm not sure why that's a requirement? To me it seems like a bonus :)

matthewbauer · 2020-04-16T21:10:14Z

Hm, I don't really want to add flags to the legacy commands (nix-instantiate, etc.). Especially when the output is hard to parse (e.g. if I understand this correctly, the output on stdout doesn't distinguish between the regular nix-instantiate output and the IFD derivations). The nix command is intended to have a --json flag whenever appropriate for non-trivial output.

Is there a nix command equivalent to nix-instantiate and nix-store? We could make this an option to nix eval, but we'd still need to get the final .drv through nix-instantiate

This keeps track of "importedDrvs" regardless of --include-ifds, so we can always add support for a new command if needed. For instance, maybe something like:

$ nix closure --dev ./default.nix

would be the equivalent of:

$ nix-store -qR --include-outputs $(nix-instantiate --include-ifd ./default.nix) | grep -v \.drv$

infinisil · 2020-04-16T22:05:14Z

Could this be used to work around #719?

Ericson2314 · 2020-04-16T22:17:14Z

Perhaps libnixexpr should store a lambda in EvalState that could do either the logging or the accumulation for #3494 or #3506. That adds the core missing feature to the libraries while punting on the user interface the executable should have.

matthewbauer · 2020-04-16T22:17:32Z

Could this be used to work around #719?

Yeah, you can register the .drv (or its output) as a gcroot I think, with something like nix-store --add-result --indirect -r $(nix-instantiate --include-ifd). gc'ing ifd actually kind of makes sense - we have no live references to the imported derivation. Perhaps we could have Nix create a "super-drv" which records what IFD happened to produce a given derivation, and an accompanying gc-keep-super-derivations = true

Ericson2314 · 2020-04-16T22:18:51Z

Yeah I would just make a little file with all the drv paths and addToStore it with the proper references.

matthewbauer · 2020-04-16T22:18:55Z

Perhaps libnixexpr should store a lambda in EvalState that could do either the logging or the accumulation for #3494 or #3506. That adds the core missing feature to the libraries while punting on the user interface the executable should have.

I don't think there's any reason you can't do both #3494 and #3506.

Ericson2314 · 2020-04-16T22:21:07Z

@matthewbauer absolutely one could do both. Why my lambda thing you would write a lambda that both accumulated and logged.

I just don't want to see nothing happen because we went back and forth on the exact user interface: clearly there is a strong desire for some way to get this information from Nix.

matthewbauer · 2020-04-16T22:25:37Z

@matthewbauer absolutely one could do both. Why my lambda thing you would write a lambda that both accumulated and logged.

I just don't want to see nothing happen because we went back and forth on the exact user interface: clearly there is a strong desire for some way to get this information from Nix.

Well it's not so useful because no one uses Nix as a library. I think it would be better to just get something working through one of the Nix commands. Adding this feature to nix is definitely doable, if only there was an equivalent to nix-instantiate!

hyperfekt · 2020-04-17T22:11:13Z

Given that this is the second instance (compare #3086) someone has taken the time to implement something like this, there seems to be a considerable need.

alyssais · 2020-04-18T13:44:24Z

Hm, I don't really want to add flags to the legacy commands (nix-instantiate, etc.).

So no new Nix features can be stabilised until the experimental CLI has been?

domenkozar · 2020-04-21T10:10:40Z

@edolstra 🙏

edolstra · 2020-04-21T21:00:21Z

The new CLI has no equivalent for nix-instantiate because it tries to hide the concept of .drv files. (This would allow switching away from having to materialize the dependency graph in the filesystem as .drv files, and instead sending an in-memory representation of the graph to the daemon, for instance.)

The old CLI is full of flags that seem "bolted on" with little structure or organization (e.g. nix-instantiate --eval does something completely different from nix-instantiate). The new CLI should avoid those mistakes. Maybe we can come up with higher-level subcommands that directly address the use cases for which --include-ifd is intended.

There is also the issue of whether we should even want to enshrine the concept of IFD in the user interface. IFD has serious issues as you probably know.

This print out all of the store paths referenced in the Nix expression. Examples provided in --help. This is useful with tools like cachix, and replace the previous usages of: $ nix-store -qR --include-outputs $(nix-instantiate '<nixpkgs>' -A hello) which now become: $ nix refs --build --run nixpkgs.hello which can be piped into cachix like: $ nix refs --build --run nixpkgs.hello | cachix push my-cache Related to cachix/cachix#52 and NixOS#3506

matthewbauer · 2020-04-22T02:12:04Z

The new CLI has no equivalent for nix-instantiate because it tries to hide the concept of .drv files. (This would allow switching away from having to materialize the dependency graph in the filesystem as .drv files, and instead sending an in-memory representation of the graph to the daemon, for instance.)

The old CLI is full of flags that seem "bolted on" with little structure or organization (e.g. nix-instantiate --eval does something completely different from nix-instantiate). The new CLI should avoid those mistakes. Maybe we can come up with higher-level subcommands that directly address the use cases for which --include-ifd is intended.

There is also the issue of whether we should even want to enshrine the concept of IFD in the user interface. IFD has serious issues as you probably know.

Yeah - avoiding .drv's would be nice. Can you take a look at #3523 which adds a nix refs command.

matthewbauer · 2020-04-24T18:24:12Z

Closing in favor of #3523

This primop lets you get information on paths realised (pathExists, import, readFile) by Nix. It tracks paths realised in Nix & puts the results under the "paths" attr. For instance: $ nix eval --impure --expr '(builtins.traceContext (import ./.).outPath)' { paths = [ "/nix/store/2mpgi4bvn8py4liv9w3mjxd2c5r7bvv8-source" "/nix/store/3k7i1rdqcgyzjxvqm37hapdidy4ls4s3-source" "/nix/store/kqxic0j6wpsaw2bb51hr1yc1nb1z2xw8-source" ]; value = "/nix/store/3k7i1rdqcgyzjxvqm37hapdidy4ls4s3-source"; } This is an alternative to the --include-eval-refs from NixOS#3523 & --include-ifd from NixOS#3506.

domenkozar requested a review from edolstra April 16, 2020 20:40

domenkozar added the improvement label Apr 18, 2020

matthewbauer mentioned this pull request Apr 22, 2020

Add --include-build-refs to `nix path-info' command #3523

Closed

matthewbauer closed this Apr 24, 2020

matthewbauer mentioned this pull request Apr 21, 2021

Add traceContext primop #4728

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --include-ifd option to nix-instantiate #3506

Add --include-ifd option to nix-instantiate #3506

matthewbauer commented Apr 16, 2020 •

edited

edolstra commented Apr 16, 2020

domenkozar commented Apr 16, 2020

matthewbauer commented Apr 16, 2020

infinisil commented Apr 16, 2020

Ericson2314 commented Apr 16, 2020

matthewbauer commented Apr 16, 2020

Ericson2314 commented Apr 16, 2020

matthewbauer commented Apr 16, 2020

Ericson2314 commented Apr 16, 2020 •

edited

matthewbauer commented Apr 16, 2020

hyperfekt commented Apr 17, 2020

alyssais commented Apr 18, 2020

domenkozar commented Apr 21, 2020 •

edited

edolstra commented Apr 21, 2020

matthewbauer commented Apr 22, 2020

matthewbauer commented Apr 24, 2020

Add --include-ifd option to nix-instantiate #3506

Add --include-ifd option to nix-instantiate #3506

Conversation

matthewbauer commented Apr 16, 2020 • edited

edolstra commented Apr 16, 2020

domenkozar commented Apr 16, 2020

matthewbauer commented Apr 16, 2020

infinisil commented Apr 16, 2020

Ericson2314 commented Apr 16, 2020

matthewbauer commented Apr 16, 2020

Ericson2314 commented Apr 16, 2020

matthewbauer commented Apr 16, 2020

Ericson2314 commented Apr 16, 2020 • edited

matthewbauer commented Apr 16, 2020

hyperfekt commented Apr 17, 2020

alyssais commented Apr 18, 2020

domenkozar commented Apr 21, 2020 • edited

edolstra commented Apr 21, 2020

matthewbauer commented Apr 22, 2020

matthewbauer commented Apr 24, 2020

matthewbauer commented Apr 16, 2020 •

edited

Ericson2314 commented Apr 16, 2020 •

edited

domenkozar commented Apr 21, 2020 •

edited