Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/nix
base: 258da08de7f0
Choose a base ref
...
head repository: NixOS/nix
compare: b10c9dd35817
Choose a head ref
  • 18 commits
  • 30 files changed
  • 3 contributors

Commits on Dec 9, 2020

  1. Revert "Re-query for the derivation outputs in the post-build-hook"

    This reverts commit 1b1e076.
    
    Using `queryPartialDerivationOutputMap` assumes that the derivation
    exists locally which isn't the case for remote builders.
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    6758e65 View commit details
    Browse the repository at this point in the history
  2. Test the post-build-hook with remote builders

    Regression test for #4245
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    ee7c94f View commit details
    Browse the repository at this point in the history
  3. Store the final drv outputs in memory when building remotely

    The `DerivationGoal` has a variable storing the “final” derivation
    output paths that is used (amongst other things) to fill the environment
    for the post build hook. However this variable wasn't set when the
    build-hook is used, causing a crash when both hooks are used together.
    
    Fix this by setting this variable (from the informations in the db) after a run
    of the post build hook.
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    c87267c View commit details
    Browse the repository at this point in the history
  4. libstore/openStore: fix stores with IPv6 addresses

    In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
    using an IPv6 address:
    
      nix ping-store --store ssh://root@2001:db8::1
    
    This is also useful for `nixops(1)` where you could specify an IPv6
    address in `deployment.targetHost`.
    
    However, this behavior is broken on `nixUnstable` and fails with the
    following error:
    
      $ nix store ping --store ssh://root@2001:db8::1
      don't know how to open Nix store 'ssh://root@2001:db8::1'
    
    This happened because `openStore` from `libstore` uses the `parseURL`
    function from `libfetchers` which expects a valid URL as defined in
    RFC2732. However, this is unsupported by `ssh(1)`:
    
      $ nix store ping --store 'ssh://root@[2001:db8::1]'
      cannot connect to 'root@[2001:db8::1]'
    
    This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
    also `root@[2001:db8::1]` since the latter one is useful to pass query
    parameters to the remote store.
    
    In order to achieve this, the following changes were made:
    
    * The URL regex from `url-parts.hh` now allows an IPv6 address in the
      form `2001:db8::1` and also `[2001:db8::1]`.
    
    * In `libstore`, a new function named `extractConnStr` ensures that a
      proper URL is passed to e.g. `ssh(1)`:
    
      * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
        the brackets will be removed using a regex. No additional validation
        is done here as only strings parsed by `parseURL` are expected.
    
      * In any other case, the string will be left untouched.
    
    * The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
      `ssh://` and `ssh-ng://`).
    
    Unresolved questions:
    
    * I'm not really sure whether we want to allow both variants of IPv6
      addresses in the URL parser. However it should be noted that both seem
      to be possible according to RFC2732:
    
      > This document incudes an update to the generic syntax for Uniform
      > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
      > for IPv6 addresses and allows the use of "[" and "]" within a URI
      > explicitly for this reserved purpose.
    
    * Currently, it's not supported to specify a port number behind the
      hostname, however it seems as this is not really supported by the URL
      parser. Hence, this is probably out of scope here.
    Ma27 committed Dec 9, 2020
    Copy the full SHA
    93a8a00 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #4319 from Ma27/store-v6-addr

    libstore/openStore: fix stores with IPv6 addresses
    edolstra committed Dec 9, 2020
    Copy the full SHA
    eb458ad View commit details
    Browse the repository at this point in the history
  6. Merge pull request #4342 from tweag/fix-remote-build-hook

    fix remote build hook
    edolstra committed Dec 9, 2020
    Copy the full SHA
    253571e View commit details
    Browse the repository at this point in the history
  7. Use no substituers by default in the tests

    Otherwise https://cache.nixos.org is chosen by default, causing the OSX
    testsuite to hang inside the sandbox.
    
    (In a way, this is probably rugging an actual bug under the carpet as
    Nix should be able to gracefully timeout in such a case, but that's
    beyond mac OSX-fu)
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    5286310 View commit details
    Browse the repository at this point in the history
  8. Merge pull request #4343 from tweag/fix-osx-ci

    Use no substituers by default in the tests
    edolstra committed Dec 9, 2020
    Copy the full SHA
    29fbc3c View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2020

  1. Add lvlNotice log level

    This is like syslog's LOG_NOTICE: "normal, but significant,
    condition".
    edolstra committed Dec 10, 2020
    Copy the full SHA
    a8f533b View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    c6a1bcd View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2020

  1. Fix the nix command with CA derivations

    Prevents a crash because most `nix` subcommands assumed that derivations
    know their output path, which isn't the case for CA derivations
    thufschmitt committed Dec 11, 2020
    Copy the full SHA
    eb45308 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command

    Fix the `nix` command with CA derivations
    edolstra committed Dec 11, 2020
    Copy the full SHA
    9c143c4 View commit details
    Browse the repository at this point in the history
  3. Store metadata about drv outputs realisations

    For each known realisation, store:
    - its output
    - its output path
    
    This comes with a set of needed changes:
    
    - New `realisations` module declaring the types needed for describing
      these mappings
    - New `Store::registerDrvOutput` method registering all the needed informations
      about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
    - new `Store::queryRealisation` method to retrieve the informations for a
      derivations
    
    This introcudes some redundancy on the remote-store side between
    `wopQueryDerivationOutputMap` and `wopQueryRealisation`.
    However we might need to keep both (regardless of backwards compat)
    because we sometimes need to get some infos for all the outputs of a
    derivation (where `wopQueryDerivationOutputMap` is handy), but all the
    stores can't implement it − because listing all the outputs of a
    derivation isn't really possible for binary caches where the server
    doesn't allow to list a directory.
    thufschmitt committed Dec 11, 2020
    Copy the full SHA
    58cdab6 View commit details
    Browse the repository at this point in the history
  4. Rework the db schema for derivation outputs

    Add a new table for tracking the derivation output mappings.
    
    We used to hijack the `DerivationOutputs` table for that, but (despite its
    name), it isn't a really good fit:
    
    - Its entries depend on the drv being a valid path, making it play badly with
      garbage collection and preventing us to copy a drv output without copying
      the whole drv closure too;
    - It dosen't guaranty that the output path exists;
    
    By using a different table, we can experiment with a different schema better
    suited for tracking the output mappings of CA derivations.
    (incidentally, this also fixes #4138)
    thufschmitt committed Dec 11, 2020
    Copy the full SHA
    3ac9d74 View commit details
    Browse the repository at this point in the history
  5. Store the realisations as JSON in the binary cache

    Fix #4332
    thufschmitt authored and Théophane Hufschmitt committed Dec 11, 2020
    Copy the full SHA
    8914e01 View commit details
    Browse the repository at this point in the history
  6. Register the outputs of non-resolved derivations

    When building a (non fully-input-addressed) derivation, we first resolve it to
    a `BasicDerivation` that depends directly on the output paths of its
    dependencies (rather than on the derivation outputs) − this is what enables
    early cutoff.
    
    Currently we register the realisation of the resolved derivation, so that
    subsequent calls to `nix-build` can directly reuse this without rebuilding the
    derivation.  However, we don't register the unresolved derivation, meaning that
    we have to re-do the resolution step each time. This in turn means that we must
    either keep all the build inputs or keep the knowledge of their output path
    (without the content of the path itself). The former is very costly in
    disk-space (it amounts to making every build-input a runtime dependency), and
    the latter comes with a big set of challenges in presence of non-determinism
    that makes this a too heavy-handed solution for the time being (though we might
    switch to it eventually if we can find a way to solve its issues as it's a
    rather tempting design).
    
    To avoid these issues, we choose a third way which is to also register the
    realisation of the unresolved derivation − so we can completely sidestep the
    resolving phase, which makes these behave as classical input-addressed
    derivations.
    thufschmitt committed Dec 11, 2020
    Copy the full SHA
    5a2aafa View commit details
    Browse the repository at this point in the history
  7. Don't crash nix-build when not all outputs are realised

    Change the `nix-build` logic for linking/printing the output paths to allow for
    some outputs to be missing. This might happen when the toplevel
    derivation didn't have to be built, either because all the required
    outputs were already there, or because they have all been substituted.
    thufschmitt committed Dec 11, 2020
    Copy the full SHA
    27905f1 View commit details
    Browse the repository at this point in the history
  8. Test the garbage collection of CA derivations

    Simple test to ensure that `nix-build && nix-collect-garbage &&
    nix-build -j0` works as it should
    thufschmitt committed Dec 11, 2020
    Copy the full SHA
    b10c9dd View commit details
    Browse the repository at this point in the history