Commits on Oct 29, 2019

1. Recursive Nix support …

   This allows Nix builders to call Nix to build derivations, with some
   limitations.
   
   Example:
   
     let nixpkgs = fetchTarball channel:nixos-18.03; in
   
     with import <nixpkgs> {};
   
     runCommand "foo"
       {
         buildInputs = [ nix jq ];
         NIX_PATH = "nixpkgs=${nixpkgs}";
       }
       ''
         hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')
   
         $hello/bin/hello
   
         mkdir -p $out/bin
         ln -s $hello/bin/hello $out/bin/hello
   
         nix path-info -r --json $hello | jq .
       ''
   
   This derivation makes a recursive Nix call to build GNU Hello and
   symlinks it from its $out, i.e.
   
     # ll ./result/bin/
     lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello
   
     # nix-store -qR ./result
     /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
     /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
     /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo
   
   This is implemented as follows:
   
   * Before running the outer builder, Nix creates a Unix domain socket
     '.nix-socket' in the builder's temporary directory and sets
     $NIX_REMOTE to point to it. It starts a thread to process
     connections to this socket. (Thus you don't need to have nix-daemon
     running.)
   
   * The daemon thread uses a wrapper store (RestrictedStore) to keep
     track of paths added through recursive Nix calls, to implement some
     restrictions (see below), and to do some censorship (e.g. for
     purity, queryPathInfo() won't return impure information such as
     signatures and timestamps).
   
   * After the build finishes, the output paths are scanned for
     references to the paths added through recursive Nix calls (in
     addition to the inputs closure). Thus, in the example above, $out
     has a reference to $hello.
   
   The main restriction on recursive Nix calls is that they cannot do
   arbitrary substitutions. For example, doing
   
     nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10
   
   is forbidden unless /nix/store/kmwd... is in the inputs closure or
   previously built by a recursive Nix call. This is to prevent
   irreproducible derivations that have hidden dependencies on
   substituters or the current store contents. Building a derivation is
   fine, however, and Nix will use substitutes if available. In other
   words, the builder has to present proof that it knows how to build a
   desired store path from scratch by constructing a derivation graph for
   that path.
   
   Probably we should also disallow instantiating/building fixed-output
   derivations (specifically, those that access the network, but
   currently we have no way to mark fixed-output derivations that don't
   access the network). Otherwise sandboxed derivations can bypass
   sandbox restrictions and access the network.
   
   When sandboxing is enabled, we make paths appear in the sandbox of the
   builder by entering the mount namespace of the builder and
   bind-mounting each path. This is tricky because we do a pivot_root()
   in the builder to change the root directory of its mount namespace,
   and thus the host /nix/store is not visible in the mount namespace of
   the builder. To get around this, just before doing pivot_root(), we
   branch a second mount namespace that shares its /nix/store mountpoint
   with the parent.
   
   Recursive Nix currently doesn't work on macOS in sandboxed mode
   (because we can't change the sandbox policy of a running build) and in
   non-root mode (because setns() barfs).

   

   edolstra committed Oct 29, 2019
   Configuration menu

   • View commit details
   • Copy full SHA for b341b5f
   • Browse repository at this point

   Copy the full SHA

   b341b5f View commit details

   Browse the repository at this point in the history

2. Use nix to cache g++ invocations …

   We call nix to run g++ on the preprocessed C++ source file. Thus
   recompilation can be avoided if the file was previously built with the
   same contents.
   
   For example:
   
     $ time make src/libstore/build.o
     CPP    src/libstore/build.o
     CXX    src/libstore/build.o
     [1 built, 0.0 MiB DL]
   
     real    0m32.560s
     user    0m0.516s
     sys     0m0.252s
   
   Then after changing a comment in build.cc:
   
     $ time make src/libstore/build.o
     CPP    src/libstore/build.o
     CXX    src/libstore/build.o
   
     real    0m0.844s
     user    0m0.514s
     sys     0m0.296s

   

   edolstra committed Oct 29, 2019
   1 Configuration menu

   • View commit details
   • Copy full SHA for 0b9e795
   • Browse repository at this point

   Copy the full SHA

   0b9e795 View commit details

   Browse the repository at this point in the history