Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/nix
base: 5bce6ad8d37a
Choose a base ref
...
head repository: NixOS/nix
compare: 4b4fc7f1e0a0
Choose a head ref
  • 10 commits
  • 22 files changed
  • 5 contributors

Commits on Dec 9, 2020

  1. Revert "Re-query for the derivation outputs in the post-build-hook" 

    This reverts commit 1b1e076.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
    @thufschmitt
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    6758e65 View commit details
    Browse the repository at this point in the history

  2. Test the post-build-hook with remote builders 

    Regression test for #4245
    @thufschmitt
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    ee7c94f View commit details
    Browse the repository at this point in the history

  3. Store the final drv outputs in memory when building remotely 

    The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
    @thufschmitt
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    c87267c View commit details
    Browse the repository at this point in the history

  4. libstore/openStore: fix stores with IPv6 addresses 

    In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
    @Ma27
    Ma27 committed Dec 9, 2020
    Copy the full SHA
    93a8a00 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #4319 from Ma27/store-v6-addr 

    libstore/openStore: fix stores with IPv6 addresses
    @edolstra
    edolstra committed Dec 9, 2020
    Copy the full SHA
    eb458ad View commit details
    Browse the repository at this point in the history

  6. Merge pull request #4342 from tweag/fix-remote-build-hook 

    fix remote build hook
    @edolstra
    edolstra committed Dec 9, 2020
    Copy the full SHA
    253571e View commit details
    Browse the repository at this point in the history

  7. Store metadata about drv outputs realisations 

    For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
    @thufschmitt
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    b62dd81 View commit details
    Browse the repository at this point in the history

  8. Rework the db schema for derivation outputs 

    Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
    @thufschmitt
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    30025cb View commit details
    Browse the repository at this point in the history

  9. Use a better prefix for the storage of realisations in binary caches

    @thufschmitt
    thufschmitt committed Dec 9, 2020
    Copy the full SHA
    ecba106 View commit details
    Browse the repository at this point in the history

  10. Register the outputs for all the built derivations 

    The requirement for having the derivation on-disk doesn't hold anymore as the new db table used doesn't require it

Co-authored-by: John Ericson <git@JohnEricson.me>
    @Ericson2314 @thufschmitt
    2 people authored and thufschmitt committed Dec 9, 2020
    Copy the full SHA
    4b4fc7f View commit details
    Browse the repository at this point in the history