`NIX_PATH`-free `nixpkgs` #30399

Gabriella439 · 2017-10-14T03:23:41Z

This adds a new section to the Nixpkgs manual documenting how to bootstrap and
pin nixpkgs without the use of the NIX_PATH

The main reasons for including this code in the documentation (as opposed to
a file in the nixpkgs repository were):

To improve discoverability of this trick
You wouldn't be able to programmatically retrieve the code anyway ☺
There's no obvious place to include this file in the repository

Credit to @taktoa who authored this code and Awake Security (that sponsored this work and
approved open sourcing it). I'm only upstreaming the work that was already done

Motivation for this change

The NIX_PATH is a common source of non-reproducibility in commercial Nix deployments.
This fixes that by not only pinning nixpkgs but doing so in such a way that the pinning
process does not itself depend on the NIX_PATH

Things done

I built this on macOS:

$ nix-build docs/

... and opened up the generated documentation to verify that the section
rendered correctly

cc: @peti and @basvandijk, who I mentioned this trick to earlier today

This adds a new section to the Nixpkgs manual documenting how to bootstrap and pin `nixpkgs` without the use of the `NIX_PATH` The main reasons for including this code in the documentation (as opposed to a file in the `nixpkgs` repository were): * To improve discoverability of this trick * You wouldn't be able to programmatically retrieve the code anyway ☺ * There's no obvious place to include this file in the repository

taktoa · 2017-10-14T05:24:06Z

doc/bootstrap.xml

+    tarball = stage1-tarball;
+  };
+
+  stage1 = import stage1-path { inherit system; };


We should probably set config = {}; in the nixpkgs arguments here, to prevent nondeterminism caused by ~/.config/nixpkgs/config.nix.

taktoa · 2017-10-14T05:25:00Z

doc/bootstrap.xml

+  stage1 = import stage1-path { inherit system; };
+};
+
+let


Why have this be in a separate let block, rather than putting it in the with block above?

taktoa · 2017-10-14T05:26:14Z

doc/bootstrap.xml

+  # ... then this file would produce this attribute set:
+  #
+  # $ nix-instantiate --eval --expr 'import ./default.nix'
+  # { 16_09 = <CODE>; 17_03 = <CODE>; 17_09 = <CODE>; }


Maybe it should be mentioned that <CODE> is a Nix path value.

vaibhavsagar · 2017-10-14T06:38:32Z

I remember hearing that fetchTarball is not cached, which is why we chose the worse NIX_PATH approach. Is this no longer the case?

Edit: I see that you're not using fetchTarball so it doesn't matter.

taktoa · 2017-10-14T07:13:24Z

@vaibhavsagar Pretty much all of this rigmarole is because we want the semantics of fetchTarball but we want it to cache properly. Once Nix 1.12 is released, fetchTarball will have a sha256 option, and most of this code will be obviated as a result.

taktoa · 2017-10-14T07:24:21Z

doc/bootstrap.xml

+        "$coreutils/mkdir" -p "$out"
+        cd "$out"
+        "$gzip" -c -d "$tarball" > temporary.tar
+        "$tar" -xf temporary.tar --strip-components=1


instead of doing this in two steps, we can combine it into one (n.b.: this code should be tested):

"$gzip" -d < "$tarball" | "$tar" -x --strip-components=1

this idea courtesy of @cleverca22's similar bootstrapping expression for IOHK

peti · 2017-10-18T15:32:41Z

I hope this is not a stupid question, but what is the advantage of this code over the following?

let
  nixpkgs = (import <nixpkgs> {}).fetchgit {
              url = "git://github.com/nixos/nixpkgs.git";
              rev = "f93a8ee1105f4cc3770ce339a8c1a4acea3b2fb6";
              sha256 = "01fnyw711p6kf9qpdabys9im10hlih1l1pxwp06wkq7b9wsljawd";
            };
in
  with import nixpkgs {};
  [...]

vaibhavsagar · 2017-10-18T15:39:12Z

@peti, what happens when you do that with NIX_PATH=""? The code in the PR doesn't expect <nixpkgs> to be available. When running builds in e.g. buildEnv, we currently need to provide some <nixpkgs> to allow bootstrapping. As mentioned above, fetchTarball is available in builtins so we could use that, except that it doesn't cache.

Gabriella439 · 2017-10-18T15:40:20Z

@peti:

Your example assumes that nixpkgs is on the NIX_PATH. The bootstrap that @taktoa wrote works even with an empty or malformed NIX_PATH
I believe you want to use fetchFromGitHub instead of fetchgit because fetchFromGitHub is faster
- fetchFromGitHub takes advantage of the fact that GitHub's API lets you download a tarball for a specific revision instead of cloning a repository
Even if you use fetchFromGitHub you assume that it will never change
- We wrote this bootstrap derivation with the assumption that the builtins and builtin-paths would be more stable and forward-compatible than fetchFromGitHub

@taktoa

... as suggested by @taktoa I also took the liberty of using long flag names for clarity

@taktoa

... as suggested by @taktoa This prevents the derivation from accidentally picking up impure `config` sources like `~/.config/nixpkgs/config.nix` or `$NIXPKGS_CONFIG`

@taktoa

... as suggested by @taktoa

cleverca22 · 2017-10-18T16:18:45Z

another benefit of allowing it to work with an empty NIX_PATH, is that you can use that empty path to audit that you have removed every impure <nixpkgs> from the project, so when NIX_PATH isnt empty, you know that it will be ignored

peti · 2017-10-18T18:21:47Z

So the assumption is that (a) the user has a working installation of Nix that is, basically, capable of compiling non-trivial software and (b) the user has a broken $NIX_PATH.

Is that combination of circumstances very likely to occur?

I mean, I see how the scheme you've come up with would work in such a situation while others, that reference <nixpkgs>, would not. I'm just not sure whether the problem this scheme fixes actually exists.

I also find some of the text misleading, i.e saying that the use of $NIX_PATH would introduce "variability" that will make your builds less reproducible feels like a bit of a stretch.

Gabriella439 · 2017-10-18T19:12:28Z

@peti: The strongest argument is the one that @cleverca22 gave. Our code base used to be fully of impure <nixpkgs> references that people would keep introducing unwittingly. We wasted a LOT of time hunting down and fixing build failures and cache misses due to these impure references. Emptying the NIX_PATH is the only way to systematically ensure that you have no such references

peti · 2017-10-18T19:24:52Z

@Gabriel439, I see your point. I am totally in favor of having that technique in our toolset, i.e. it's pretty cool that it's possible to use a deterministic version of Nixpkgs without having Nixpkgs available already.

I just feel like the documentation this PR offers doesn't represent the respective pros and cons very accurately. I, for one, could not see from the text why this kind of elaborate scheme might be desirable and what it achieves.

Gabriella439 · 2017-10-18T19:25:52Z

Alright, I'll update the documentation to reflect this discussion

edolstra · 2017-10-19T09:54:21Z

I think this might be better as a wiki article. By putting it in the manual, we're elevating it to something like a best current practice, and I don't think there is a consensus on that. Also, it seems strange to me to distribute non-trivial bits of source code via the manual for users to cut & paste verbatim.

BTW, @peti's example in #30399 (comment) can be changed to use builtins.fetchurlto fetch from GitHub, or (in Nix 1.12) use builtins.fetchgit. That removes the dependency on <nixpkgs>.

domenkozar · 2017-10-21T10:32:17Z

There are two downsides to this approach:

it doesn't work on non-Nix bootstrapped Nix (for example Nix installed from gentoo)
it needs bootstrap script in each repo

We use something like the following: https://github.com/input-output-hk/iohk-ops/blob/develop/fetch-nixpkgs.nix

All of this will be superseeded by Nix 1.12, so it would be good to know if that's coming soon before merging this.

Gabriella439 · 2017-10-21T19:36:37Z

@domenkozar: How will this be superseded by Nix 1.12?

taktoa · 2017-10-22T03:34:16Z

@Gabriel439 From my comment above: "Once Nix 1.12 is released, fetchTarball will have a sha256 option, and most of this code will be obviated as a result."

Gabriella439 · 2017-10-23T16:07:26Z

@edolstra: Oh, for some reason I thought the wiki was shut down in favor of the manual. I didn't realize that the Wiki was revived. Then I can move this example to the Wiki instead. I'll also incorporate the feedback to use the builtins to fetch the tarball instead of fetchFromGitHub from nixpkgs

However, I want to point out that you have to copy and paste the bootstrap script by necessity. If there were a reproducible way to fetch the bootstrap script from the nixpkgs repository then you wouldn't need the script in the first place

Gabriella439 · 2017-10-23T16:33:05Z

@edolstra: So after digging into the builtins suggestions, all three of fetchTarball, builtins.fetchgit, and builtins.fetchurl do not work in restricted mode, so they fail on Hydra if I remember correctly. So I'm not sure that Nix 1.12 will solve these issues for us and we may still have to stick with fetchFromGitHub

taktoa · 2017-10-23T16:34:14Z

@Gabriel439 Presumably builtins.fetchTarball should be allowed to work in restricted mode as long as the sha256 parameter is given. If not, I think that is definitely worth changing.

Gabriella439 · 2017-10-23T16:36:06Z

@taktoa: Yeah, you're correct. It will not require restricted mode if you provide the hash:

https://github.com/NixOS/nix/blob/e2f9a61dc95f8f0ee51995ad88f4c5d5bd0e7740/src/libexpr/primops.cc#L1940

So I think then pre-Nix-1.12 fetchFromGitHub is the best solution and post-Nix-1.12 fetchTarball with the expected hash is the best solution

taktoa · 2017-10-23T16:43:49Z

A couple other suggestions regarding this Nix script:

We should probably make it futureproof using the builtins.nixVersion attribute to warn users that are on Nix 1.12 of the superior version they could be using.
Maybe we should simplify it as much as possible; the current version includes extra features that won't be relevant to every user (ideally, I think, we'd be giving the source of a file called fetchFromGitHub.nix such that import fetchFromGitHub.nix is equivalent to fetchFromGitHub). The part where we fetch the stage 2 nixpkgs is not actually relevant to the utility of the script.

Gabriella439 · 2017-10-23T17:15:04Z

@taktoa: I like that idea. I'll refactor this to be a fetchNixpkgs.nix bootstrap script which expects just a revision and hash. I hope people don't mind if I continue to use this pull request for now until people agree with the final state and then I'll submit the change to the Wiki

Gabriella439 · 2017-10-23T17:52:04Z

@taktoa: Here's the simpler version I've got after your feedback:

# This file provides a way to fetch `nixpkgs` with an empty `NIX_PATH`.  This
# comes in handy if you want to remove impure references to the `NIX_PATH` from
# your code base
#
# To use this code, save this to a `fetchNixpkgs.nix` file within your project
# and then referencing this file like so:
#
# ```nix
# let
#   fetchNixpkgs = import ./fetchNixpkgs.nix;
#
#   nixpkgs = fetchNixpkgs {
#      rev = "76d649b59484607901f0c1b8f737d8376a904019";
#
#      sha256 = "01c2f4mj4ahir0sxk9kxbymg2pki1pc9a3y6r9x6ridry75fzb8h";
#   };
#
#   pkgs = import nixpkgs {};
#
# in
#   ...
# ```

{ rev, sha256 }:

with rec {
  system = builtins.currentSystem;

  tarball = import <nix/fetchurl.nix> {
    url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
    inherit sha256;
  };

  script = builtins.toFile "nixpkgs-unpacker" ''
    "$coreutils/mkdir" "$out"
    cd "$out"
    "$gzip" --decompress < "$tarball" | "$tar" --extract --strip-components=1
  '';

  builtin-paths = import <nix/config.nix>;

  nixpkgs = builtins.derivation {
    name = "nixpkgs-${builtins.substring 0 6 rev}";

    builder = builtin-paths.shell;

    args = [ script ];

    inherit tarball system;

    inherit (builtin-paths) tar gzip coreutils;
  };
};

nixpkgs

The only thing missing is suggesting the user to upgrade to a better script for Nix 1.12 and later, mainly because I'd like to first complete the better version of the script so that the suggestion can link to it

taktoa · 2017-10-23T17:58:36Z

You missed a semicolon in the comment at the top (after the closing brace before pkgs = ...).

The version for Nix 1.12 would probably look like

{ rev, sha256 }:

builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
  inherit sha256;
}

taktoa · 2017-10-23T18:05:20Z

It just occurred to me: since @edolstra doesn't want to backport the sha256 that was added to builtins.fetchTarball in 1.12, what if we just somehow replaced the implementation of fetchTarball in Nix 1.11 with the code that uses <nix>? Then you could just do:

assert (
  ((builtins.compareVersion "1.11.16" builtins.nixVersion) >= 0)
  || ((builtins.match "^1[.]12.*$" builtins.nixVersion) != null));

{ rev, sha256 }:

builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
  inherit sha256;
}

and it would work on both Nix 1.11 and 1.12.

EDIT: actually, in light of the fact that the <nix> solution fails in the case where Nix was built from source, I think the ideal option is probably just for someone to backport the code.

basvandijk · 2017-10-23T18:08:55Z

I really like the ideas put forth in this conversation.

One trick that I think is worth mentioning as well is to patch the pinned nixpkgs with a list of patches. Patching is usually less work than rebasing all those commits on some branch you have to maintain.

For example this is the nixpkgs.nix expression we use at LumiGuide (I should rewrite this to use the ideas in the PR so that I don't depend on the NIX_PATH):

# This expression returns the nix/store path to our version of nixpkgs.
# It ensures that all engineers use the same revision of nixpkgs.
# It is used in the following files:
#
# * ./default.nix:
#   import (import ./nixpkgs.nix) { ... }
#
# Note that this expression requires that "nixpkgs" can be found in the
# NIX_PATH. However, since this expression defines the path to nixpkgs we have a
# bootstrapping problem. To bootstrap a workstation we will create an install-cd
# which has a copy of nixpkgs in its nix/store and set its NIX_PATH to that
# store path.
#
# We also have the bootstrapping problem on hydra.lumi.guide. Here we "solve" it
# by adding nixpkgs as an input like:
#
#   nixpkgs "Git checkout" https://github.com/LumiGuide/nixpkgs.git f806ddf
#
# This technique was inspired by the article:
#
#   Reproducible Development Environments by Rok Garbas
#   https://garbas.si/2015/reproducible-development-environments.html

let pkgs = import <nixpkgs> {};
    nixpkgsVersion = pkgs.lib.importJSON ./nixpkgs-version.json;

    nixpkgs = pkgs.fetchFromGitHub {
       owner   = "NixOS";
       repo    = "nixpkgs";
       rev     = nixpkgsVersion.rev;
       sha256  = nixpkgsVersion.sha256;
     };

    patches = [
      # elasticsearch-5.4
      (pkgs.fetchpatch {
        url = "https://github.com/LumiGuide/nixpkgs/commit/7ad19877d41592d45ed8723818238d1c17550644.patch";
        sha256 = "15j2pgvp577ykbf6zvx57kygpfbmp8n7nwyi7pizcix4hasz46bx";
      })

      # Kibana-5.4, logstash-5.4 and beats-5.4
      (pkgs.fetchpatch {
        url = "https://github.com/LumiGuide/nixpkgs/commit/c0f9714afd7be6e045252d88c485eef9a5278129.patch";
        sha256 = "089b8pb02dx390mh3g135d80hclg9lz6xwqyd5ivcbrxqchh18wd";
      })

      # ELK: 5.4.0 -> 5.4.2
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/946952a10891c6e6764abf546cc8cc1f3f54f668.patch";
        sha256 = "15a37ydlnqvsjr3r0351h3x4zk579i62dh6qf002f7dzr2lf0rrs";
      })

      # ELK: 5.4.2 -> 5.5.2
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/09bde4a2cdeed27dca8a145502d405d4c986a304.patch";
        sha256 = "088x4rm5gh31pncmjdvrws07nixdnbybb9ydmn35ldsyzdnalkp3";
      })

      # ELK: 5.5.2 -> 5.6.1
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/ea8e7ed1e32b40fc90173e35d5b8afe747d84e38.patch";
        sha256 = "1js6cg6gik42j86ljmh6dfgsqpnzipkyx4jmzq5mb7qv8dh8rcqq";
      })

      # journalbeat: 5.1.2 -> 5.4.1
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/766cbda1f32112633f265d8ad0aad3252346dfb0.patch";
        sha256 = "1mq3rhxyf15h29n9z94zb83n70lg6s7ywqrrbskqx10kv2ha4asj";
      })

      # journalbeat: 5.4.1 -> 5.5.0
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/4a93ce179dc0026b5c021b9fe6aa89b93527bba5.patch";
        sha256 = "0hlirh46rpq8p5j29syii38yam98x79vx2h4yfxkahf7yz8mj6nw";
      })

      # journalbeat: 5.5.0 -> 5.6.0
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/8d07b99b96661a60e14141824bf891e10776cf85.patch";
        sha256 = "0qs2a30i5p2m683a4wbmxwg7272am1ssbib0p1rmkmxprv4maik4";
      })

      # Fixes SSL dependency that the central server needs
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/7c0f6f4be53453e08a1d4a27cd40a46c4e920ce8.patch";
        sha256 = "1r2nm3x5w9rg9772865i228rag7shigfw0da7hcwd1vh019wgk6x";
      })

      # Adds the postage package and NixOS module
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/943c78b10d8ed4418dbb6fb9a89e6f416af511d5.patch";
        sha256 = "13vhrkihbw7nrwplxlhfvwm493h53y7yzs8j5nsxnhv70hhpiwc4";
      })

      # postage: 3.2.17 -> 3.2.18
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/d7f2c1f2b81a8b1c15cbe3d234cf8abc73bc5a2f.patch";
        sha256 = "1limynv4aj4i3z704x14a38g5x73xakdpswf8h4amy6rlrmg3n12";
      })

      # Adds flockit
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/872d8fc5c9f4f8b5d31cb5a2a886e986dc0abd52.patch";
        sha256 = "0r9gc97mjq1maxica8gyiikd48cy5mm2b59mgp5h6p489alndjnb";
      })

      # strongswan: 5.5.3 -> 5.6.0
      (pkgs.fetchpatch {
        url = "https://github.com/NixOS/nixpkgs/commit/a8868900ab2f063005d812a484e22c8c6f3b4364.patch";
        sha256 = "0z8v45wxgwjl35ydgr4vsbbn38k80cjsaqhwgz80zv0kipxzkldl";
      })
    ];

in pkgs.runCommand ("nixpkgs-" + nixpkgsVersion.rev) {inherit nixpkgs patches; } ''
  cp -r $nixpkgs $out
  chmod -R +w $out
  for p in $patches ; do
    echo "Applying patch $p"
    patch -d $out -p1 < "$p"
  done
''

Gabriella439 · 2017-10-23T18:15:45Z

@taktoa: Instead of an assertion, why not use an if expression predicated on the Nix version. Nix is lazy, so it would only evaluate builtins.fetchTarball if the version was sufficiently high. I verified that this works for Nix 1.11.4 on my machine:

# This file provides a way to fetch `nixpkgs` with an empty `NIX_PATH`.  This
# comes in handy if you want to remove impure references to the `NIX_PATH` from
# your code base
#
# To use this code, save this to a `fetchNixpkgs.nix` file within your project
# and then referencing this file like so:
#
# ```nix
# let
#   fetchNixpkgs = import ./fetchNixpkgs.nix;
#
#   nixpkgs = fetchNixpkgs {
#      rev = "76d649b59484607901f0c1b8f737d8376a904019";
#
#      sha256 = "01c2f4mj4ahir0sxk9kxbymg2pki1pc9a3y6r9x6ridry75fzb8h";
#   }
#
#   pkgs = import nixpkgs {};
#
# in
#   ...
# ```

{ rev, sha256 }:

if 0 <= builtins.compareVersions builtins.nixVersion "1.12"
then

builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
  inherit sha256;
}

else

# `builtins.fetchTarball` only accepts a `sha256` argument in Nix version 1.12
# or later
with rec {
  system = builtins.currentSystem;

  tarball = import <nix/fetchurl.nix> {
    url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
    inherit sha256;
  };

  script = builtins.toFile "nixpkgs-unpacker" ''
    "$coreutils/mkdir" "$out"
    cd "$out"
    "$gzip" --decompress < "$tarball" | "$tar" --extract --strip-components=1
  '';

  builtin-paths = import <nix/config.nix>;

  nixpkgs = builtins.derivation {
    name = "nixpkgs-${builtins.substring 0 6 rev}";

    builder = builtin-paths.shell;

    args = [ script ];

    inherit tarball system;

    inherit (builtin-paths) tar gzip coreutils;
  };
};

nixpkgs

Also, it's not clear why you have the more complex version test

@basvandijk: I can also to add additional documentation for how to patch nixpkgs, but I consider that orthogonal to how to fetch nixpkgs

taktoa · 2017-10-23T18:20:52Z

@Gabriel439
The code with the assert was intended for a hypothetical future version of Nix 1.11 in which the fetchTarball command works as it does in Nix 1.12. I don't know if we should be promulgating code that is designed to handle versions of Nix 1.11 other than the most recent version.

The more complex version test is because of the following issue with the nixUnstable prerelease:

nix-repl> builtins.nixVersion
"1.12pre5619_346aeee1"
nix-repl> builtins.compareVersions "1.12" "1.12.1
-1
nix-repl> builtins.compareVersions "1.12" builtins.nixVersion
1

Gabriella439 · 2017-10-23T18:34:41Z

@taktoa: I think it's better to be conservative about when we enable the fetchTarball path, even if that means that we miss a few cases like 1.12pre5619_346aeee1 or a hypothetical 1.11.* version that supports fetchTarball with a hash. More generally, I'd prefer to recommend code that is slightly less optimal and always succeeds rather than slightly more optimal but sometimes fails in certain corner cases. These little corner case failures add up in the Nix ecosystem and ruin the user experience so I don't want to add to the pile

The code is sufficiently self-describing that users can easily delete the pre-Nix-1.12 fallback case if they care enough

taktoa · 2017-10-23T18:35:30Z

@Gabriel439
Alright, that makes sense I guess.

From some conversation with @cleverca22 I have been informed that the strings contained in <nix/config.nix> are just bare strings that happen to coincide with store paths, so the code that we currently have here should break under Nix sandboxing (and neither of us know why it doesn't seem to break). The fix is to call builtins.storePath to add a "context" to the strings; this has been done in the IOHK version of the code, and here's a modified version of what you have that incorporates the changes:

# This file provides a way to fetch `nixpkgs` with an empty `NIX_PATH`.  This
# comes in handy if you want to remove impure references to the `NIX_PATH` from
# your code base
#
# To use this code, save this to a `fetchNixpkgs.nix` file within your project
# and then referencing this file like so:
#
# ```nix
# let
#   fetchNixpkgs = import ./fetchNixpkgs.nix;
#   nixpkgs = fetchNixpkgs {
#      rev = "76d649b59484607901f0c1b8f737d8376a904019";
#      sha256 = "01c2f4mj4ahir0sxk9kxbymg2pki1pc9a3y6r9x6ridry75fzb8h";
#   };
#   pkgs = import nixpkgs { config = {}; };
# in ...
# ```

{ rev, sha256 }:

if 0 <= builtins.compareVersions builtins.nixVersion "1.12"
then

builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
  inherit sha256;
}

else

# `builtins.fetchTarball` only accepts a `sha256` argument in Nix version or later
with rec {
  system = builtins.currentSystem;

  tarball = import <nix/fetchurl.nix> {
    url = "https://github.com/NixOS/nixpkgs/archive/${rev}.tar.gz";
    inherit sha256;
  };

  builtin-paths = import <nix/config.nix>;

  nixpkgs = builtins.derivation {
    name = "nixpkgs-${builtins.substring 0 6 rev}";

    builder = builtins.storePath builtin-paths.shell;

    args = [ 
      (builtins.toFile "nixpkgs-unpacker" ''
        "$coreutils/mkdir" "$out"
        cd "$out"
        "$gzip" --decompress < "$tarball" | "$tar" --extract --strip-components=1
      '')
    ];

    inherit tarball system;

    tar = builtins.storePath builtin-paths.tar;
    gzip = builtins.storePath builtin-paths.gzip;
    coreutils = builtins.storePath builtin-paths.coreutils;
  };
};

nixpkgs

EDIT: forgot to add builtins.storePath to the builder = builtin-paths.shell;

Gabriella439 · 2017-10-23T18:44:12Z

Alright, I'll use the version with builtins.storePath. I'll wait a day and if there's no other feedback then I'll submit this to the Wiki

domenkozar · 2017-10-30T11:42:39Z

The most minimal version I could come up with for nixcon2017 (so it fits into the slides) is:

let
  spec = builtins.fromJSON (builtins.readFile ./nixpkgs-src.json);
  src = import <nix/fetchurl.nix> {
    url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
    inherit (spec) sha256;
  };
  nixcfg = import <nix/config.nix>;
in builtins.derivation {
  system = builtins.currentSystem;
  name = "${src.name}-unpacked";
  builder = builtins.storePath nixcfg.shell;
  inherit src;
  args = [
    (builtins.toFile "builder" ''
      $coreutils/mkdir $out
      cd $out
      $gzip -d < $src | $tar -x --strip-components=1
    '')
  ];
  coreutils = builtins.storePath nixcfg.coreutils;
  tar = builtins.storePath nixcfg.tar;
  gzip = builtins.storePath nixcfg.gzip;
}

and nixpkgs-src.json:

{
    "owner":  "NixOS",
    "repo":   "nixpkgs",
    "rev": "6549173ae5dfece3c7f315185209cf4a54c2d257",
    "sha256": "1lir7n1adyv8cm4g6c18qf4c2lmwcz085ra814m12hjgywp1i3my"
}

I don't think conditionalizing on 1.12 is a good idea, best to just rewrite to hash usage once 1.12 is out a few weeks.

taktoa · 2017-10-30T18:29:29Z

@domenkozar not that it matters much, but there is a subtle bug in that code:

...
  builder = nixcfg.shell;
...

should be

...
  builder = builtins.storePath nixcfg.shell;
...

I think the only reason it works without that is because of the /bin/sh Nix impurity.

domenkozar · 2017-10-30T18:48:48Z

Thanks!

Gabriella439 · 2017-11-03T16:52:25Z

Alright, I just added this to the NixOS wiki:

How to fetch Nixpkgs with an empty NIX PATH

... and I added it to the Nix, Nixpkgs, and Cookbook categories for discoverability

domenkozar · 2017-11-06T17:34:44Z

@Gabriel439 would you also consider adding how to use it with bash scripts to set NIX_PATH if needed to pinned version? It's useful for interactive sessions.

$ cat scripts/set_nixpath.sh 
update_NIX_PATH() {
  local readlink=$(nix-instantiate --eval -E "/. + (import <nix/config.nix>).coreutils")/readlink
  local scriptDir=$(dirname -- "$($readlink -f -- "${BASH_SOURCE[0]}")")
  export NIX_PATH="nixpkgs=$(nix-build "${scriptDir}/../fetch-nixpkgs.nix" -o nixpkgs)"
}
update_NIX_PATH

Usage:

source scripts/set_nixpath.sh`

Gabriella439 · 2017-11-06T17:45:28Z

@domenkozar: Why not just:

$ export NIX_PATH=nixpkgs=$(nix-build --no-out-link ./nixpkgs.nix)

... where ./nixpkgs.nix is:

import ./fetchNixpkgs.nix { rev = "..."; sha256 = "..."; }

domenkozar · 2017-11-06T17:48:08Z

This assumes ./nixpkgs.nix is always relative to your current folder. Above script makes sure it works in any directory (and darwin with $readlink from coreutils).

Gabriella439 · 2017-11-06T17:50:57Z

@domenkozar: I still don't see the benefit. Using your solution you would still need the correct path to the set_nixpath.sh script itself, so why not instead provide the correct path to ./nixpkgs.nix

cleverca22 · 2017-11-06T18:00:33Z

BASH_SOURCE is the path to the current bash script, and the readlink -f turns that into an absolute path

Gabriella439 · 2017-11-06T18:08:55Z

@cleverca22: That's still pushing the problem one level up: you need the correct path to the script that sources the set_nixpath.sh script itself

Also, stepping back, why do we even need to set the NIX_PATH in the first place? Isn't the whole point of this change to not depend on the NIX_PATH at all?

cleverca22 · 2017-11-06T18:10:34Z

yeah, i would rather just import fetch-nixpkgs.nix from another nix file (where relative paths work without any fuss) and then just not refer to NIX_PATH at all

domenkozar · 2017-11-06T18:13:31Z

@Gabriel439 because in real world you want to override for example https://github.com/NixOS/nixops/blob/master/release.nix#L7

Gabriella439 · 2017-11-06T18:31:01Z

My intuition here is still to keep as much logic out of Bash as possible and to move the logic into Nix

cleverca22 · 2017-11-06T18:32:01Z

patching nixops to accept a pkgs set would allow this to work with an empty NIX_PATH and no bash logic

cleverca22 · 2017-11-16T04:32:30Z

NixOS/nixops@8389ead and nixops has been patched

👍

bhipple · 2018-04-29T21:59:50Z

For anyone coming late to this discussion and just wondering what the simplest way to achieve this with nix2 is, all you need to do is put these couple lines at the top of your nix file (no other files, overlays, NIX_PATH, or other dependencies required).

$ cat demo.nix
with import (builtins.fetchTarball {
  url = "https://github.com/nixos/nixpkgs/archive/91e58ad48ae5f6d64d3989f7ca133fe90a9b20cc.tar.gz";
  sha256 = "12nxpqxwxip13ydawk7awabimdj56g8dy984lbl8ip916qvkd9y0";
}) {};

# Rest of the nix expression as normal, for example:
{
  demo = buildEnv {
    name = "demo";
    paths = [ emacs tmux ripgrep ];
  };
}

which will now work with nix build -f demo.nix, no NIX_PATH or channels required.

taktoa reviewed Oct 14, 2017

View reviewed changes

Gabriel Gonzalez added 4 commits October 18, 2017 08:59

Remove unnecessary temporary.tar from stage1-path derivation

a79684f

... as suggested by @taktoa I also took the liberty of using long flag names for clarity

Set config explicitly to {}

d997aad

... as suggested by @taktoa This prevents the derivation from accidentally picking up impure `config` sources like `~/.config/nixpkgs/config.nix` or `$NIXPKGS_CONFIG`

Use with consistently

142e034

... as suggested by @taktoa

Improve documentation

3a53c2f

domenkozar mentioned this pull request Nov 1, 2017

Documentation is not clear on how to fix nixpkgs version NixOS/nixops#452

Open

Gabriella439 closed this Nov 3, 2017

Gabriella439 deleted the gabriel/nixpath_free_nixpkgs branch November 3, 2017 16:52

This was referenced Apr 27, 2019

nix-shell fails tweag/capability#69

Closed

Pin nixpkgs in nix2 style tweag/capability#70

Merged

danbst mentioned this pull request Aug 6, 2019

Running nixos-rebuild switch twice can produce different results due to dependence on current system state #65803

Open

fabaff mentioned this pull request May 2, 2024

NixOS for Pentesting Overview #81418

Open

NIX_PATH-free nixpkgs #30399

NIX_PATH-free nixpkgs #30399

Conversation

Gabriella439 commented Oct 14, 2017

Motivation for this change

Things done

taktoa Oct 14, 2017

Choose a reason for hiding this comment

taktoa Oct 14, 2017

Choose a reason for hiding this comment

taktoa Oct 14, 2017

Choose a reason for hiding this comment

vaibhavsagar commented Oct 14, 2017 • edited

taktoa commented Oct 14, 2017

taktoa Oct 14, 2017

Choose a reason for hiding this comment

peti commented Oct 18, 2017 • edited

vaibhavsagar commented Oct 18, 2017

Gabriella439 commented Oct 18, 2017

cleverca22 commented Oct 18, 2017

peti commented Oct 18, 2017

Gabriella439 commented Oct 18, 2017

peti commented Oct 18, 2017

Gabriella439 commented Oct 18, 2017

edolstra commented Oct 19, 2017

domenkozar commented Oct 21, 2017

Gabriella439 commented Oct 21, 2017

taktoa commented Oct 22, 2017

Gabriella439 commented Oct 23, 2017

Gabriella439 commented Oct 23, 2017

taktoa commented Oct 23, 2017

Gabriella439 commented Oct 23, 2017

taktoa commented Oct 23, 2017

Gabriella439 commented Oct 23, 2017

Gabriella439 commented Oct 23, 2017 • edited

taktoa commented Oct 23, 2017

taktoa commented Oct 23, 2017 • edited

basvandijk commented Oct 23, 2017

Gabriella439 commented Oct 23, 2017 • edited

taktoa commented Oct 23, 2017

Gabriella439 commented Oct 23, 2017 • edited

taktoa commented Oct 23, 2017 • edited

Gabriella439 commented Oct 23, 2017

domenkozar commented Oct 30, 2017 • edited

taktoa commented Oct 30, 2017

domenkozar commented Oct 30, 2017

Gabriella439 commented Nov 3, 2017

domenkozar commented Nov 6, 2017

Gabriella439 commented Nov 6, 2017 • edited

domenkozar commented Nov 6, 2017

Gabriella439 commented Nov 6, 2017

cleverca22 commented Nov 6, 2017

Gabriella439 commented Nov 6, 2017

cleverca22 commented Nov 6, 2017

domenkozar commented Nov 6, 2017

Gabriella439 commented Nov 6, 2017

cleverca22 commented Nov 6, 2017

cleverca22 commented Nov 16, 2017

bhipple commented Apr 29, 2018 • edited

`NIX_PATH`-free `nixpkgs` #30399

`NIX_PATH`-free `nixpkgs` #30399

vaibhavsagar commented Oct 14, 2017 •

edited

peti commented Oct 18, 2017 •

edited

Gabriella439 commented Oct 23, 2017 •

edited

taktoa commented Oct 23, 2017 •

edited

Gabriella439 commented Oct 23, 2017 •

edited

Gabriella439 commented Oct 23, 2017 •

edited

taktoa commented Oct 23, 2017 •

edited

domenkozar commented Oct 30, 2017 •

edited

Gabriella439 commented Nov 6, 2017 •

edited

bhipple commented Apr 29, 2018 •

edited