mkDerivation: Introduce .inputDerivation for shell.nix build convenience #95536

infinisil · 2020-08-15T21:39:49Z

This allows easy building and distributing of all derivations needed to
enter a nix-shell with

nix-build shell.nix -A inputDerivation

Or with cachix:

nix-build shell.nix -A inputDerivation | cachix push $name

In more technical terms: This introduces the .inputDerivation attribute on all derivations created with mkDerivation. This is another derivation that can always build successfully and whose runtime dependencies are the build time dependencies of the original derivation.

Motivation for this change

The recent blog post by @fzakaria (https://fzakaria.com/2020/08/11/caching-your-nix-shell.html) mentions:

tl;dr; you can use the following invocation to cache your nix-shell

$ nix-store --query --references $(nix-instantiate shell.nix) | \
   xargs nix-store --realise | \
   xargs nix-store --query --requisites | \
   cachix push your_cache

However after testing this I noticed that this caches more than needed due to multiple outputs not being handled separately. Since I had such an inputDerivation in the repo, I was able to compare it directly: Above command resulted in 653 paths, whereas inputDerivation only needed 475.

So I'm now upstreaming this such that it can be used by everybody. During my couple years in the #nixos IRC channel this kind of functionality was requested more than once, and with this PR there's finally a good builtin solution for it.

Ping @grahamc who once shared an original version of this
Ping @Ericson2314 @matthewbauer as people who are often involved with make-derivation.nix

And ping @zupo from Niteo, who sponsored this PR :)

Things done

fzakaria · 2020-08-15T21:53:39Z

pkgs/stdenv/generic/make-derivation.nix

+             _derivation_original_builder = derivationArg.builder;
+             _derivation_original_args = derivationArg.args;
+             builder = stdenv.shell;
+             args = [ "-c" " export > $out" ];


might be nice for those reading the code-base/pr to explain this line.

export or env (my preferred) when called with 0 arguments dumps out all current environment variables.
The end result of this derivation is a file whose contents are the environment variables.
nix-shell works mainly by placing all buildInputs into the subshell's $PATH variable.
They are therefore present in plaintext in the file and can then be found by nix when grepping for runtime dependencies.

Good point! I chose export because it's a bash builtin. I added an explanation of how it works

👍
I'm still somewhat new to Nix ~6months (non-NixOS user); and a big fan of comments with rationale :)

fzakaria · 2020-08-15T22:04:24Z

@infinisil can you elaborate on

However after testing this I noticed that this caches more than needed due to multiple outputs not being handled separately. Since I had such an inputDerivation in the repo, I was able to compare it directly: Above command resulted in 653 paths, whereas inputDerivation only needed 475.

Are you referring to derivations that produce multiple outputs ?
https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/multiple-outputs.sh

It's not clear what you mean by "not being handled separately"; can you explain it in turns of the graph-closures?

infinisil · 2020-08-15T22:21:05Z

@fzakaria Here's an example I made up, shell.nix in this PR's root:

with import ./. {};

let
  dep = runCommand "dep" {
    outputs = [ "out" "bin" ];
  } ''
    mkdir -p $out $bin/bin
    echo out > $out/out
    echo bin > $bin/bin/bin
  '';
in

stdenv.mkDerivation {
  name = "test";
  buildInputs = [ dep.out ];
}

Then the inputDerivation doesn't contain the bin output, while the nix-store --references query does:

$ nix-store -qR $(nix-build shell.nix -A inputDerivation) | grep dep
/nix/store/lgs9qbx2y3qrgdhr1ch3ndc86r55wf80-dep

$ nix-store --query --references $(nix-instantiate shell.nix) | \
  xargs nix-store --realise | \
  xargs nix-store --query --requisites | \
  grep dep
/nix/store/djxdfp8ir3921yqxbgqgfr62i8dyxvz6-dep-bin
/nix/store/lgs9qbx2y3qrgdhr1ch3ndc86r55wf80-dep

This introduces the .inputDerivation attribute on all derivations created with mkDerivation. This is another derivation that can always build successfully and whose runtime dependencies are the build time dependencies of the original derivation. This allows easy building and distributing of all derivations needed to enter a nix-shell with nix-build shell.nix -A inputDerivation

fzakaria · 2020-08-15T22:58:19Z

I checked what it would produce from the example given in my blog post

let nixpkgs = import <nixpkgs> {};
in
with nixpkgs;
with stdenv;
with stdenv.lib;
mkShell {
  name = "example-shell";
  buildInputs = [chromium];
}

$ nix-store -qR $(nix-build --no-out-link -I nixpkgs=/home/fmzakari/code/github.com/NixOS/nixpkgs shell.nix -A inputDerivation)  | wc -l

> 218

218 is even smaller than 221 than from my blog post; (unsure if it's due to changes to the closure from HEAD on nixpkgs)

👍 So this looks good to me for a solution; although I really dislike the interface / discoverability.
I like that it's more thorough of a solution; especially given multiple-outputs.

Can we include a friendlier CLI param for nix-shell that performs -A inputDerivation ?
nix-shell <FILE> --list-transitive-dependencies|--list-immediate-dependencies

Also 👍 would be a matching PR in https://github.com/NixOS/nix to augment the documentation with this new option.
AFAIK only documentation resides at https://docs.cachix.org/faq.html#is-there-a-way-to-cache-nix-shell
(ping @domenkozar )

infinisil · 2020-08-16T12:39:09Z

Discoverability isn't great right now yeah, but I'm thinking we could let this change cook a bit in nixpkgs to see how well it works before introducing any nix-shell option and co.

fzakaria · 2020-08-16T17:37:18Z

Sounds good to me and the PR looks good.

Let's open a corresponding Nix issue to track and reference once merged.

Thanks for the contribution.
Now I'll have to update my blog :)

fzakaria · 2020-08-17T21:37:14Z

Let's get this 👶 merged in.

infinisil · 2020-08-17T21:46:02Z

It would be nice for @Ericson2314 or @matthewbauer to review this, or maybe @edolstra also has something to say regarding this. I'd say let's wait a couple days for that

Mic92 · 2020-08-24T11:27:22Z

Is this not already covered in a more precise way by doing:

nix-build $(nix eval --raw --impure --expr '((import ./. {}).drvPath)')

You can than upload the derivation path instead of inputDerivation.
You can even do:

$ nix-shell $(nix eval --raw --impure --expr '((import ./. {}).drvPath)')

or

$ nix-build $(nix eval --raw --impure --expr '((import ./. {}).drvPath)') 
$ nix-shell ./result
$ echo ./result | cachix push $name

infinisil · 2020-08-24T12:54:55Z

@Mic92 Nah that's something different. What you're proposing seems to be just to nix-build the derivation (the nix eval ... command seems to be just nix-instantiate, and nix-building that is the same as nix-building the file directly). But that's not always possible (e.g. mkShell can't be built), but also the result of such a build won't depend on all build inputs. As an example, this shell.nix:

with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "test";
  unpackPhase = ":";
  dontBuild = true;
  buildInputs = [ hello ];
  installPhase = "touch $out";
}

Building that and looking at the things it depends on gives you:

$ nix-store -qR $(nix-build shell.nix)
/nix/store/qflg8wf892kgvxnn1s5bashrc6g1vc1y-test

So it doesn't depend on anything, consequently hello wouldn't be pushed to cachix.

Compare that with inputDerivation:

$ nix-store -qR $(nix-build default.nix -A inputDerivation)
/nix/store/r2nywq3ziag55zi6dqcxkpb6yla044kq-libunistring-0.9.10
/nix/store/arb8311fjm1dsrbsy8j7pdanwnz1qwxv-libidn2-2.3.0
/nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31
[...]
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
/nix/store/ab1pfk338f6gzpglsirxhvji4g9w558i-hello-2.10
/nix/store/lk7r5rifwz6hhfsc0xqj88gg4ggnrxvx-test

This contains hello and everything else needed to start a nix-shell

Mic92 · 2020-08-25T14:05:21Z

@Mic92 Nah that's something different. What you're proposing seems to be just to nix-build the derivation (the nix eval ... command seems to be just nix-instantiate, and nix-building that is the same as nix-building the file directly). But that's not always possible (e.g. mkShell can't be built), but also the result of such a build won't depend on all build inputs. As an example, this shell.nix:
with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "test";
  unpackPhase = ":";
  dontBuild = true;
  buildInputs = [ hello ];
  installPhase = "touch $out";
}
Building that and looking at the things it depends on gives you:
$ nix-store -qR $(nix-build shell.nix)
/nix/store/qflg8wf892kgvxnn1s5bashrc6g1vc1y-test
So it doesn't depend on anything, consequently hello wouldn't be pushed to cachix.

Compare that with inputDerivation:
$ nix-store -qR $(nix-build default.nix -A inputDerivation)
/nix/store/r2nywq3ziag55zi6dqcxkpb6yla044kq-libunistring-0.9.10
/nix/store/arb8311fjm1dsrbsy8j7pdanwnz1qwxv-libidn2-2.3.0
/nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31
[...]
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
/nix/store/ab1pfk338f6gzpglsirxhvji4g9w558i-hello-2.10
/nix/store/lk7r5rifwz6hhfsc0xqj88gg4ggnrxvx-test
This contains hello and everything else needed to start a nix-shell

The following works for me:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

mkShell {
  buildInputs = [ hello ];
}

$ nix-store -qR $(nix eval --raw --impure --expr '((import ./shell.nix {}).drvPath)') 
/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv
/nix/store/03f77phmfdmsbfpcc6mspjfff3yc9fdj-setup-hook.sh
/nix/store/064jmylcq7h6fa5asg0rna9awcqz3765-0001-x86-Properly-merge-GNU_PROPERTY_X86_ISA_1_USED.patch
/nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv
/nix/store/bzq60ip2z5xgi7jk6jgdw8cngfiwjrcm-bootstrap-tools.tar.xz.drv
/nix/store/wzdwpgqf2384hr2npma78mqillg5lv08-unpack-bootstrap-tools.sh
/nix/store/0zhkga32apid60mm7nh92z2970im5837-bootstrap-tools.drv
/nix/store/2adm9mf4gs34vbn1js65v90yq6v9i12p-attr-2.4.48.tar.gz.drv
/nix/store/34vpnpik7b1fb91gwv8pfw7s7v8yir01-gettext-0.21.tar.gz.drv
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
...
/nix/store/xdvhdjkpbdikmyarlhipyzxxlpiw9dra-nix-shell.drv

The derivation file always depends on all build inputs.

fzakaria · 2020-08-25T17:33:44Z

@Mic92 that's actually still incorrect & also very unfriendly (not very developer ergonomic) to boot!
Here is the example @infinisil linked from my blog post

let nixpkgs = import <nixpkgs> {};
in
with nixpkgs;
with stdenv;
with stdenv.lib;
mkShell {
  name = "example-shell";
  buildInputs = [];
  TEST="${chromium}";
}

Ideally, the closure should only be what is required by chromium
Using @infinisil changes (or my other recommended approach mentioned on the blog post) gives a closure of ~200 packages.

$ nix-store -qR $(nix-build --no-out-link -I nixpkgs=/home/fmzakari/code/github.com/NixOS/nixpkgs shell.nix -A inputDerivation)  | wc -l

> 218

If however I run your sample, I get 1656 packages
(btw I could not get nix eval to work; I had to use nix repl)

$ nix repl
nix-repl> chromium = (import ./shell.nix).drvPath
nix-repl> :p chromium
"/nix/store/2vfizyrcd0i6f9x6faz5j7grq255bxrx-example-shell.drv"

$ nix-store -qR /nix/store/2vfizyrcd0i6f9x6faz5j7grq255bxrx-example-shell.drv  | wc -l
1656

Mic92 · 2020-08-25T18:23:20Z

@Mic92 that's actually still incorrect & also very unfriendly (not very developer ergonomic) to boot!
Here is the example @infinisil linked from my blog post
let nixpkgs = import <nixpkgs> {};
in
with nixpkgs;
with stdenv;
with stdenv.lib;
mkShell {
  name = "example-shell";
  buildInputs = [];
  TEST="${chromium}";
}
Ideally, the closure should only be what is required by chromium
Using @infinisil changes (or my other recommended approach mentioned on the blog post) gives a closure of ~200 packages.
$ nix-store -qR $(nix-build --no-out-link -I nixpkgs=/home/fmzakari/code/github.com/NixOS/nixpkgs shell.nix -A inputDerivation)  | wc -l

> 218
If however I run your sample, I get 1656 packages
(btw I could not get nix eval to work; I had to use nix repl)
$ nix repl
nix-repl> chromium = (import ./shell.nix).drvPath
nix-repl> :p chromium
"/nix/store/2vfizyrcd0i6f9x6faz5j7grq255bxrx-example-shell.drv"

$ nix-store -qR /nix/store/2vfizyrcd0i6f9x6faz5j7grq255bxrx-example-shell.drv  | wc -l
1656

Yes. The drv file seems to capture more dependencies than inputDerivation which makes me wonder if inputDerivation not misses dependencies. The nix eval example uses nixUnstable flags. In nix stable I think you need to drop --raw --impure --expr.

fzakaria · 2020-08-25T18:29:36Z

@Mic92 I think the difference which I noted in the blog post; is that when doing performing a nix-store --query --requisites on a .drv; the result is the transitive closure for a source deployment for all buildInputs

This is actually not the desired outcome when caching the buildInputs; rather we want the immediate build-time dependencies of our derivation (mkShell) but for each dependency, only include their run-time dependencies.

Mic92 · 2020-08-26T18:49:51Z

@Mic92 I think the difference which I noted in the blog post; is that when doing performing a nix-store --query --requisites on a .drv; the result is the transitive closure for a source deployment for all buildInputs

This is actually not the desired outcome when caching the buildInputs; rather we want the immediate build-time dependencies of our derivation (mkShell) but for each dependency, only include their run-time dependencies.

Ok. Thanks for clarification.

nixos-discourse · 2020-08-28T19:24:22Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-export-nix-shell-dependency-for-deploy/6539/3

infinisil · 2020-08-30T17:55:49Z

I'd say let's merge this for now, as it will be pretty useful for many people, and it's a very simple addition. Any objections @Mic92 or anybody else?

Mic92 · 2020-08-30T18:18:36Z

I'd say let's merge this for now, as it will be pretty useful for many people, and it's a very simple addition. Any objections @Mic92 or anybody else?

no.

edolstra · 2020-08-31T14:15:31Z

The name inputDerivation is also not very descriptive. It's not very clear what an "input derivation" is and how it relates to shells.

infinisil · 2020-08-31T15:50:40Z

@edolstra

It's yet another undiscoverable "magic" mechanism like .override and .overrideDerivation.

I think it's justified for this case, because the alternative of it being something like pkgs.makeInputDerivation <drv> would require changes to every projects shell.nix file in order for them to use this feature. And such an inputDerivation concept is very much tied to the derivation itself, every derivation has it.

It's yet another derivation attribute that prevents all derivation inputs from being garbage-collected. It doesn't make the GC situation worse in itself but it's another GC-preventing feature that we can never get rid of.

I see this inputDerivation feature as only a temporary solution, because ideally it would be implemented in Nix itself (it has all the info to do that). Maybe via something like nix-shell <file> --list-immediate-dependencies as suggested by @fzakaria. If that's done, we can get rid of this in nixpkgs with an inputDerivation = throw "Use nix-shell --list-immediate-dependencies instead". For now though, this is the next best way to do this without requiring a Nix change.

The name inputDerivation is also not very descriptive. It's not very clear what an "input derivation" is and how it relates to shells.

I'm open to a better name. I chose inputDerivation because it's the "Derivation that depends on all build inputs of another derivation". Maybe one of:

buildableShell (because it's essentially a nix-shell environment you can build)
inputClosure/buildInputClosure because it's the closure of all input derivations, though their closure isn't really an important part
derivationInputPropagator/buildInputPropagator because it propagates build inputs to runtime dependencies

Any other/better ideas?

fzakaria · 2020-08-31T15:56:48Z

I'm also happy to contribute to documentation for the new attribute unless the plan is to make progress on something more concrete.

Should mkShell default to building the inputDerivation?
That would alleviate the "hidden" aspect of it.

A really clean user experience would be to allow the following and have it return a dummy output maybe.

nix-build shell.nix

infinisil · 2020-09-01T18:34:00Z

@fzakaria Having docs for this would be awesome!

Should mkShell default to building the inputDerivation?
That would alleviate the "hidden" aspect of it.

This would be pretty inconsistent, because you'd have to decide between nix-build shell.nix and nix-build shell.nix -A inputDerivation depending on whether the expression uses mkShell or not.

A really clean user experience would be to allow the following and have it return a dummy output maybe.
nix-build shell.nix

This can't work because shell.nix isn't any different than any other Nix file. There's nothing that could automatically transform that command to nix-build shell.nix -A inputDerivation, while not doing the same for every other nix-build command. Also note that nix-shell falls back to default.nix if it can't find a shell.nix.

fzakaria · 2020-09-01T19:26:59Z

@infinisil ah; I guess I meant instead of mkShell failing to build (through it's custom builder); have it just build inputDerivation and maybe do something extremely trivial echo $SHELL_NAME > $out

I'll work on docs.

fzakaria · 2020-09-02T19:16:52Z

@infinisil i started with a wiki entry for now
https://nixos.wiki/wiki/Caching_nix_shell_build_inputs

edolstra · 2020-09-02T19:28:48Z

By the way, how does this differ from the derivation built by nix develop? E.g.

$ nix develop --profile ./profile -c 'true'
$ head ./profile
declare -x ACLOCAL_PATH="/nix/store/9004si374yl96j6jyfgjydqh7l6frjfz-bison-3.5.1/share/aclocal:...
declare -x AR="ar"
declare -x AS="as"
...

This derivation is constructed from the original derivation here: https://github.com/NixOS/nix/blob/94a043ff3b0e9de92bdb62372f9a82186d8e871c/src/nix/develop.cc#L111-L143

You can subsequently enter a saved shell by doing nix develop ./profile BTW.

fzakaria · 2020-09-02T19:53:23Z

@edolstra i'm not on beta or don't have nix develop enabled on my nix installation so I couldn't tell you.

According to the comment to the line you linked

/* Given an existing derivation, return the shell environment as
   initialised by stdenv's setup script. We do this by building a
   modified derivation with the same dependencies and nearly the same
   initial environment variables, that just writes the resulting
   environment to a file and exits. */

It might be similar since it's ultimately "building" the derivation; whereas the mkShell fails to build (on purpose).
This would allow a runtime-closure query as opposed to compile-closure.

I think @infinisil approach is quite elegant; my only concern is documentation and exposing it through a more friendly UI in the CLI.

Mic92 · 2020-09-03T03:45:46Z

By the way, how does this differ from the derivation built by nix develop? E.g.
$ nix develop --profile ./profile -c 'true'
$ head ./profile
declare -x ACLOCAL_PATH="/nix/store/9004si374yl96j6jyfgjydqh7l6frjfz-bison-3.5.1/share/aclocal:...
declare -x AR="ar"
declare -x AS="as"
...
This derivation is constructed from the original derivation here: NixOS/nix@94a043f/src/nix/develop.cc#L111-L143

You can subsequently enter a saved shell by doing nix develop ./profile BTW.

Is this safe to use? I don't see a garbage collection root created for --profile, which is what I fixed in direnv-nix in my use flake implementation. Apart from that one could do cachix push ./profile afterwards to achieve the same as .inputDerivation. The other remaining issue is that it is not easy to integrate in let's say hydra.

edolstra · 2020-09-03T09:37:01Z

I thought it was registered as a GC root but this turned out not to be the case. I've fixed this (NixOS/nix@b07167b).

nixos-discourse · 2020-12-27T11:42:24Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/trick-to-utilize-binary-cache-in-flake-devshell/10672/3

nh2 · 2020-12-31T16:45:58Z

I found this very useful to cache developer environments.

We use mkShell to make the environments with python Packages in as well as other programs on PATH. We also have the need to ensure on CI that they work, and to add GC roots to those environments so that they don't get accidentally GCd.

I am using now:

{
  python-env = pkgs.mkShell {
    buildInputs = [
      (python.withPackages (ps: with ps; [
        numpy
        scipy
        # ... other stuff
      ]))
      exiftool
    ];
  };
  python-env-noenv = python-env.inputDerivation;
}

This allows me to

nix-shell -A python-env

to enter the shell, and

nix-build -A python-env-noenv --out-link my-gc-root

to do the CI and the GC root.

nixos-discourse · 2021-01-21T23:01:47Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-do-i-access-the-devshell-outpath-from-inside-flake-nix-to-bundle-cache-it/11089/5

nixos-discourse · 2021-02-28T06:09:55Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/getting-unique-id-from-nix-expression-to-avoid-redundant-build/11716/3

nixos-discourse · 2021-05-04T02:45:15Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/devshell-inputderivation-flake-example/12878/1

infinisil · 2021-09-17T15:17:36Z

An interesting further use case for this is discovering why a certain derivation is a build input:

$ nix-store -q --tree $(nix-build -A hello.inputDerivation)
/nix/store/67bf3xlwpd8c96y4rxsmnbsr761vysm4-hello-2.10
+---/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
+---/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
+---/nix/store/dpjnjrqbgbm8a5wvi1hya01vd8wyvsq4-bash-4.4-p23
|   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47
|   |   +---/nix/store/js5xyvpmw5x6n29b2cg6k0gh1nxaw98j-libidn2-2.3.2
|   |   |   +---/nix/store/7v6ck5q9f540qsni7pkc9pwk1yd9yygn-libunistring-0.9.10
|   |   |   |   +---/nix/store/7v6ck5q9f540qsni7pkc9pwk1yd9yygn-libunistring-0.9.10 [...]
|   |   |   +---/nix/store/js5xyvpmw5x6n29b2cg6k0gh1nxaw98j-libidn2-2.3.2 [...]
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   +---/nix/store/dpjnjrqbgbm8a5wvi1hya01vd8wyvsq4-bash-4.4-p23 [...]
+---/nix/store/r4bl79l2bdjawmr2rhhqvci56qh0fkvv-stdenv-linux
|   +---/nix/store/1570w56jrkvr90w9x158vyb5vahnk18v-coreutils-8.32
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/i0hmd3b0m5w501cdq1sx7ip19ba8mc26-attr-2.5.1
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/i0hmd3b0m5w501cdq1sx7ip19ba8mc26-attr-2.5.1 [...]
|   |   +---/nix/store/pni52adzp1cq83syj5wy1rgallv4nary-acl-2.3.1
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/i0hmd3b0m5w501cdq1sx7ip19ba8mc26-attr-2.5.1 [...]
|   |   |   +---/nix/store/pni52adzp1cq83syj5wy1rgallv4nary-acl-2.3.1 [...]
|   |   +---/nix/store/1570w56jrkvr90w9x158vyb5vahnk18v-coreutils-8.32 [...]
|   +---/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh
|   +---/nix/store/2wn093wbc6ps4brcsppxjd14vxvaa8a2-gawk-5.1.0
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/2wn093wbc6ps4brcsppxjd14vxvaa8a2-gawk-5.1.0 [...]
|   +---/nix/store/435paza0j48aa9vgvf6r2l12nrg4ld11-patchelf-0.12
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/0kiykyrnrpfhmjwxwx89kxr20hmf5304-gcc-10.3.0-lib
|   |       +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |       +---/nix/store/0kiykyrnrpfhmjwxwx89kxr20hmf5304-gcc-10.3.0-lib [...]
|   +---/nix/store/dpjnjrqbgbm8a5wvi1hya01vd8wyvsq4-bash-4.4-p23 [...]
|   +---/nix/store/v0slhpb2y3xa7gmv4q6gblkdk7n0f09j-gnugrep-3.6
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/6jr0k1iz0iv2y339qmbbs9hj1l1qif5y-pcre-8.44
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/6jr0k1iz0iv2y339qmbbs9hj1l1qif5y-pcre-8.44 [...]
|   |   +---/nix/store/v0slhpb2y3xa7gmv4q6gblkdk7n0f09j-gnugrep-3.6 [...]
|   +---/nix/store/4xs1xyj8728yvh9y5v6ji819kwgfy2fv-gcc-wrapper-10.3.0
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/0kiykyrnrpfhmjwxwx89kxr20hmf5304-gcc-10.3.0-lib [...]
|   |   +---/nix/store/1570w56jrkvr90w9x158vyb5vahnk18v-coreutils-8.32 [...]
|   |   +---/nix/store/1vb8jqkiw6scxn7ncjh1hw72hfnr2nr0-expand-response-params
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/6z35qvn00xrjvaznv9kfy8xddbbdz4gl-glibc-2.33-47-bin
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/6z35qvn00xrjvaznv9kfy8xddbbdz4gl-glibc-2.33-47-bin [...]
|   |   +---/nix/store/dpjnjrqbgbm8a5wvi1hya01vd8wyvsq4-bash-4.4-p23 [...]
|   |   +---/nix/store/iwd8ic5hhwdxn5dga0im55g5hjl270cd-glibc-2.33-47-dev
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/6z35qvn00xrjvaznv9kfy8xddbbdz4gl-glibc-2.33-47-bin [...]
|   |   |   +---/nix/store/ph2k4vnsa1zfxlckrmfg997byhbm0z17-linux-headers-5.12
|   |   +---/nix/store/29bjq5hw1qglybp1a5f3ij9gxc2fyf94-binutils-wrapper-2.35.1
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/1570w56jrkvr90w9x158vyb5vahnk18v-coreutils-8.32 [...]
|   |   |   +---/nix/store/1vb8jqkiw6scxn7ncjh1hw72hfnr2nr0-expand-response-params [...]
|   |   |   +---/nix/store/6z35qvn00xrjvaznv9kfy8xddbbdz4gl-glibc-2.33-47-bin [...]
|   |   |   +---/nix/store/dpjnjrqbgbm8a5wvi1hya01vd8wyvsq4-bash-4.4-p23 [...]
|   |   |   +---/nix/store/iwd8ic5hhwdxn5dga0im55g5hjl270cd-glibc-2.33-47-dev [...]
|   |   |   +---/nix/store/v8imx1nvyz0hgvx9cbcmh6gp4ngw3ffj-binutils-2.35.1
|   |   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   |   +---/nix/store/0kiykyrnrpfhmjwxwx89kxr20hmf5304-gcc-10.3.0-lib [...]
|   |   |   |   +---/nix/store/dc3r2nbhckwn41hq5vaxxbj7z0lqdwb1-zlib-1.2.11
|   |   |   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   |   +---/nix/store/v8imx1nvyz0hgvx9cbcmh6gp4ngw3ffj-binutils-2.35.1 [...]
|   |   |   +---/nix/store/29bjq5hw1qglybp1a5f3ij9gxc2fyf94-binutils-wrapper-2.35.1 [...]
|   |   +---/nix/store/dlni53myj53kx20pi4yhm7p68lw17b07-gcc-10.3.0
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/0kiykyrnrpfhmjwxwx89kxr20hmf5304-gcc-10.3.0-lib [...]
|   |   |   +---/nix/store/dc3r2nbhckwn41hq5vaxxbj7z0lqdwb1-zlib-1.2.11 [...]
|   |   |   +---/nix/store/iwd8ic5hhwdxn5dga0im55g5hjl270cd-glibc-2.33-47-dev [...]
|   |   |   +---/nix/store/dlni53myj53kx20pi4yhm7p68lw17b07-gcc-10.3.0 [...]
|   |   +---/nix/store/v0slhpb2y3xa7gmv4q6gblkdk7n0f09j-gnugrep-3.6 [...]
|   |   +---/nix/store/4xs1xyj8728yvh9y5v6ji819kwgfy2fv-gcc-wrapper-10.3.0 [...]
|   +---/nix/store/5bxrjkyvqmzn1p897652y3lwa9fxagpw-gnutar-1.34
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/pni52adzp1cq83syj5wy1rgallv4nary-acl-2.3.1 [...]
|   |   +---/nix/store/5bxrjkyvqmzn1p897652y3lwa9fxagpw-gnutar-1.34 [...]
|   +---/nix/store/7jk6k46f56rszzc1bxi8mdrvcw53pym4-xz-5.2.5-bin
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/nq5q61hm0r9y5f215nkmm1gjkmha8jkq-xz-5.2.5
|   |       +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |       +---/nix/store/nq5q61hm0r9y5f215nkmm1gjkmha8jkq-xz-5.2.5 [...]
|   +---/nix/store/8zxndz5ag0p6s526c2xyllhk1nrn4c3i-audit-tmpdir.sh
|   +---/nix/store/aicl3kwfnaizk45aygm8bviqv7lk0a16-patch-2.7.6
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/znbsgj80bhyd8clfy1k0k667nxqvxy7m-ed-1.17
|   |   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   |   +---/nix/store/znbsgj80bhyd8clfy1k0k667nxqvxy7m-ed-1.17 [...]
|   |   +---/nix/store/aicl3kwfnaizk45aygm8bviqv7lk0a16-patch-2.7.6 [...]
|   +---/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh
|   +---/nix/store/c8n9kcdddp9np665xz6ri61b383nxvz8-move-systemd-user-units.sh
|   +---/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh
|   +---/nix/store/cl3qd985p1yxyfkj96v0hqxiy3w69xq5-compress-man-pages.sh
|   +---/nix/store/fyaryjvghbkpfnsyw97hb3lyb37s1pd6-move-lib64.sh
|   +---/nix/store/gm2w08wwsa3vd500d8vq879s2lv65ldh-gnused-4.8
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/gm2w08wwsa3vd500d8vq879s2lv65ldh-gnused-4.8 [...]
|   +---/nix/store/h54dzwd7rdh2jlcv91424csl6d0ccgjy-strip.sh
|   +---/nix/store/ih2zkh2mbrx2c766ryk2i9hhlkly7snr-bzip2-1.0.6.0.2-bin
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/spymc49wl8fr71ysjiq3q0jivc7mbja9-bzip2-1.0.6.0.2
|   |       +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |       +---/nix/store/spymc49wl8fr71ysjiq3q0jivc7mbja9-bzip2-1.0.6.0.2 [...]
|   +---/nix/store/ihxk2vlm0vi7c4j3gpm084kbxvz6v585-findutils-4.8.0
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/1570w56jrkvr90w9x158vyb5vahnk18v-coreutils-8.32 [...]
|   |   +---/nix/store/ihxk2vlm0vi7c4j3gpm084kbxvz6v585-findutils-4.8.0 [...]
|   +---/nix/store/kd4xwxjpjxi71jkm6ka0np72if9rm3y0-move-sbin.sh
|   +---/nix/store/kjx1mv85c5cgsrr4bwar22j7hbwj834m-diffutils-3.7
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/1570w56jrkvr90w9x158vyb5vahnk18v-coreutils-8.32 [...]
|   |   +---/nix/store/kjx1mv85c5cgsrr4bwar22j7hbwj834m-diffutils-3.7 [...]
|   +---/nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh
|   +---/nix/store/liva1jnjdskrn57s42kfawr2zz66szzm-gzip-1.10
|   |   +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|   |   +---/nix/store/liva1jnjdskrn57s42kfawr2zz66szzm-gzip-1.10 [...]
|   +---/nix/store/mjjy30kxz775bhhi6j9phw81qh6dsbrf-move-docs.sh
|   +---/nix/store/ngg1cv31c8c7bcm2n8ww4g06nq7s4zhm-set-source-date-epoch-to-latest.sh
|   +---/nix/store/pr6nzbscdpg94yvr151lrif2kg0csx7b-reproducible-builds.sh
|   +---/nix/store/pvkiiw0mp1yx0b21b7k105yywccychgh-gnumake-4.3
|       +---/nix/store/9bh3986bpragfjmr32gay8p95k91q4gy-glibc-2.33-47 [...]
|       +---/nix/store/pvkiiw0mp1yx0b21b7k105yywccychgh-gnumake-4.3 [...]
+---/nix/store/67bf3xlwpd8c96y4rxsmnbsr761vysm4-hello-2.10 [...]

In comparison to nix-store -q --tree $(nix-instantiate -A hello), this tree really only contains the packages needed for the build of that derivation, and is therefore also much smaller. Can also use

$ nix why-depends $(nix-build -A hello.inputDerivation) /nix/store/nq5q61hm0r9y5f215nkmm1gjkmha8jkq-xz-5.2.5
/nix/store/67bf3xlwpd8c96y4rxsmnbsr761vysm4-hello-2.10
╚═══/: …".declare -x stdenv="/nix/store/r4bl79l2bdjawmr2rhhqvci56qh0fkvv-stdenv-linux".declare -x strict…
    => /nix/store/r4bl79l2bdjawmr2rhhqvci56qh0fkvv-stdenv-linux
    ╚═══setup: …v7lk0a16-patch-2.7.6 /nix/store/7jk6k46f56rszzc1bxi8mdrvcw53pym4-xz-5.2.5-bin".defaultNativeBuil…
        => /nix/store/7jk6k46f56rszzc1bxi8mdrvcw53pym4-xz-5.2.5-bin
        ╚═══bin/lzmadec: …LIBC_2.4.GLIBC_2.3.4./nix/store/nq5q61hm0r9y5f215nkmm1gjkmha8jkq-xz-5.2.5/lib:/nix/store>
            => /nix/store/nq5q61hm0r9y5f215nkmm1gjkmha8jkq-xz-5.2.5

When I designed `mkShell`, I didn't have a good idea of what the output should look like and so decided to make the build fail. In practice, this causes quite a bit of confusion and complications because now the shell cannot be part of a normal package set without failing the CI as well. This commit changes that build phase to record all the build inputs in a file. That way it becomes possible to build it, makes sure that all the build inputs get built as well, and also can be used as a GC root. (by applying the same trick as NixOS#95536). The documentation has also been improved to better describe what mkShell does and how to use it.

When I designed `mkShell`, I didn't have a good idea of what the output should look like and so decided to make the build fail. In practice, this causes quite a bit of confusion and complications because now the shell cannot be part of a normal package set without failing the CI as well. This commit changes that build phase to record all the build inputs in a file. That way it becomes possible to build it, makes sure that all the build inputs get built as well, and also can be used as a GC root. (by applying the same trick as #95536). The documentation has also been improved to better describe what mkShell does and how to use it.

rnhmjoj · 2022-05-04T08:11:42Z

This attribute is very useful, but due to the way it's implemented it misses some dependencies.
If you pinned Nixpkgs and are going offline for some time, you have to manually add the Nixpkgs tarball (to nativeBuildInputs or an environment variable) to save it from the garbage collection.

infinisil · 2022-05-16T19:01:51Z

@rnhmjoj I believe that's part of this bigger problem: NixOS/nix#719. Also it depends on whether pkgs.fetch* or builtins.fetch* is used to get nixpkgs, because with the former the caching happens with the Nix store, whereas with the latter the caching happens additionally with ~/.cache/nix. Not sure how it all plays together at this moment.

Same rationale as upstream, see NixOS/nixpkgs#95536

infinisil requested review from Ericson2314 and matthewbauer as code owners August 15, 2020 21:39

fzakaria reviewed Aug 15, 2020

View reviewed changes

infinisil force-pushed the inputDerivation branch from 369225e to e841464 Compare August 15, 2020 22:02

infinisil force-pushed the inputDerivation branch from e841464 to eb8e69c Compare August 15, 2020 22:11

infinisil force-pushed the inputDerivation branch from eb8e69c to 3e1a40d Compare August 15, 2020 22:24

ofborg bot added 6.topic: stdenv Standard environment 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Aug 15, 2020

domenkozar approved these changes Aug 24, 2020

View reviewed changes

infinisil merged commit 9114979 into NixOS:master Aug 31, 2020

infinisil deleted the inputDerivation branch August 31, 2020 13:46

domenkozar mentioned this pull request Dec 2, 2020

Mention inputDerivation for nix-shell FAQ cachix/docs.cachix.org#22

Closed

zimbatm mentioned this pull request Jan 2, 2022

mkShell: make it buildable #153194

Merged

13 tasks

amarshall added a commit to amarshall/devshell that referenced this pull request Sep 30, 2022

Add inputDerivation

917a891

Same rationale as upstream, see NixOS/nixpkgs#95536

amarshall mentioned this pull request Oct 1, 2022

Add inputDerivation to enable binary caching numtide/devshell#226

Closed

infinisil mentioned this pull request Dec 16, 2022

provide getDerivationEnvironment as a primop to reference nix develop drv from Nix code NixOS/nix#7468

Open

fricklerhandwerk mentioned this pull request Dec 19, 2022

nix-shell dependencies can be garbage collected any time now / persistent nix-shell envs NixOS/nix#2208

Closed

amarshall added a commit to amarshall/devshell that referenced this pull request Mar 2, 2023

Add inputDerivation

8e150a1

Same rationale as upstream, see NixOS/nixpkgs#95536

infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023

mkDerivation: Introduce .inputDerivation for shell.nix build convenience #95536

mkDerivation: Introduce .inputDerivation for shell.nix build convenience #95536

Conversation

infinisil commented Aug 15, 2020

Motivation for this change

Things done

fzakaria Aug 15, 2020

Choose a reason for hiding this comment

infinisil Aug 15, 2020 • edited

Choose a reason for hiding this comment

fzakaria Aug 15, 2020

Choose a reason for hiding this comment

fzakaria commented Aug 15, 2020

infinisil commented Aug 15, 2020

fzakaria commented Aug 15, 2020 • edited

infinisil commented Aug 16, 2020

fzakaria commented Aug 16, 2020

fzakaria commented Aug 17, 2020

infinisil commented Aug 17, 2020

Mic92 commented Aug 24, 2020 • edited

infinisil commented Aug 24, 2020

Mic92 commented Aug 25, 2020 • edited

fzakaria commented Aug 25, 2020 • edited

Mic92 commented Aug 25, 2020

fzakaria commented Aug 25, 2020 • edited

Mic92 commented Aug 26, 2020

nixos-discourse commented Aug 28, 2020

infinisil commented Aug 30, 2020

Mic92 commented Aug 30, 2020

edolstra commented Aug 31, 2020

infinisil commented Aug 31, 2020

fzakaria commented Aug 31, 2020

infinisil commented Sep 1, 2020

fzakaria commented Sep 1, 2020

fzakaria commented Sep 2, 2020

edolstra commented Sep 2, 2020 • edited

fzakaria commented Sep 2, 2020

Mic92 commented Sep 3, 2020 • edited

edolstra commented Sep 3, 2020

nixos-discourse commented Dec 27, 2020

nh2 commented Dec 31, 2020

nixos-discourse commented Jan 21, 2021

nixos-discourse commented Feb 28, 2021

nixos-discourse commented May 4, 2021

infinisil commented Sep 17, 2021

rnhmjoj commented May 4, 2022

infinisil commented May 16, 2022

infinisil Aug 15, 2020 •

edited

fzakaria commented Aug 15, 2020 •

edited

Mic92 commented Aug 24, 2020 •

edited

Mic92 commented Aug 25, 2020 •

edited

fzakaria commented Aug 25, 2020 •

edited

fzakaria commented Aug 25, 2020 •

edited

edolstra commented Sep 2, 2020 •

edited

Mic92 commented Sep 3, 2020 •

edited