Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

installer: simplify the per-build installation #4224

Merged
merged 1 commit into from Nov 21, 2020

Conversation

zimbatm
Copy link
Member

@zimbatm zimbatm commented Nov 6, 2020

The goal is to allow the installation and testing of arbitrary Nix
versions. Extend the base installer to accept a --tarball-url-prefix <url> to
change where the Nix tarball is getting downloaded from.

Once this is merged it should allow to:

  1. Pick an evaluation at https://hydra.nixos.org/jobset/nix/master that
    looks healthy
  2. Select the installedScript build and find the store path.

Now equipped with all of this, use an instance of nar-serve to fetch the
install script and release tarballs:

curl -sfL https://nar-serve.numtide.com/nix/store/rkv4yh7pym941bhj0849zqdkg2546bdv-installer-script/install \
  | sh --tarball-url-prefix https://nar-serve.numtide.com/nix/store

Or with cachix, strip the /nix/store and derivation name and then:

curl -sfL https://mycache.cachix.org/serve/rkv4yh7pym941bhj0849zqdkg2546bdv/install \
  | sh --tarball-url-prefix https://mycache.cachix.org/serve

Fixes #4047

@zimbatm
Copy link
Member Author

zimbatm commented Nov 6, 2020

This should work in theory. It's a little bit hard for me to test this as it requires all the architectures to produce the installer script.

@zimbatm
Copy link
Member Author

zimbatm commented Nov 6, 2020

If that approach is not satisfactory, my other proposal would be to generate a second installer script that downloads the tarballs from https://nar-serve.numtide.com/

@matthewbauer
Copy link
Member

So having to set NIX_RELEASE_URL is annoying, but this is definitely an improvement over not being able to easily install Nix via Hydra builds since the script hardcodes releases.nixos.org.

The main concern I have is that the hash is only going to match if NIX_RELEASE_URL is the same as the install script's Nix. Perhaps it shouldn't error if a custom NIX_RELEASE_URL is provided but the hash doesn't match.

I was brainstorming some alternatives that don't require users to set NIX_RELEASE_URL. It's kind of tricky - we know the hash of the tarball, so we could get the NAR of the Nix tarball from hydra.nixos.org, but that's in NAR, so we can't really extract it without a working Nix. Alternatively, we could upload all of the Hydra to tarballs.nixos.org, and just use that, but it only works if you are the official Hydra, so not so great.

A long term goal, I think, would be to bootstrap Nix with a statically built Nix binary. This installer would do the same stuff as the existing shell script, but also be embedded with a Nix expression to build itself dynamically. If that Nix expression is cached, it would be downloaded, otherwise Nix can build itself. This avoids the issue of relying on being in some official Nix release. But this is of course out of scope for this PR.

@edolstra
Copy link
Member

edolstra commented Nov 9, 2020

Some issues:

  • Environment variables like NIX_RELEASE_URL are not discoverable.

  • It's not very user-friendly that you have to figure out two URLs and they have to correspond (e.g. https://hydra.nixos.org/build/1234/download/1/install and https://hydra.nixos.org/build/5678/download/1/XXX).

  • Hydra is not very suitable as a release server. For instance, builds could be deleted from the database, store paths could be garbage collected. Also Hydra is slow.

Maybe a better approach is to install a stable Nix and then do nix-env -i /nix/store/... to install an unstable version from the binary cache.

@zimbatm
Copy link
Member Author

zimbatm commented Nov 9, 2020

There are some overlapping but distinct ideas:
a. Make it easier to test the install script. I'm thinking in particular the macOS one. In that case, I think it's okay to get temporary URLs that might break, even if it's a bit tricky.
b. Bootstrap Nix to any version. In that case, Eelco's approach of installing the stable Nix and then realizing the unstable one is okay. It would be nice if there was a one-shot approach available.

For NIX_RELEASE_URL, it could be replaced by a command-line argument.

What other ways forward is there?

@zimbatm
Copy link
Member Author

zimbatm commented Nov 9, 2020

Here is another proposal. It allows us to use nar-serve (cf commit message).

@zimbatm
Copy link
Member Author

zimbatm commented Nov 13, 2020

installer: allow nar-serve installations

The goal is to allow the installation and testing of arbitrary Nix
versions. Extend the base installer to accept a --tarball-url <url> to
change where the Nix tarball is getting downloaded from.

Once this is merged it should allow to:

  1. Pick an evaluation at https://hydra.nixos.org/jobset/nix/master that
    looks healthy
  2. Select the installedScript build and find the store path.

Now equipped with all of this, use an instance of nar-serve to fetch the
install script and release tarballs:

curl -sfL https://nar-serve.numtide.com/nix/store/rkv4yh7pym941bhj0849zqdkg2546bdv-installer-script/install \
  | sh --tarball-url https://nar-serve.numtide.com/nix/store

Or with cachix:

curl -sfL https://mycache.cachix.org/serve/rkv4yh7pym941bhj0849zqdkg2546bdv/install \
  | sh --tarball-url https://mycache.cachix.org/serve

@zimbatm
Copy link
Member Author

zimbatm commented Nov 17, 2020

fixed the merge conflict with the addition of Darwin.arm64

scripts/install.in Outdated Show resolved Hide resolved
scripts/install.in Outdated Show resolved Hide resolved
@zimbatm zimbatm force-pushed the per-build-installer branch 5 times, most recently from dcee08c to 276063c Compare November 21, 2020 15:38
scripts/install.in Outdated Show resolved Hide resolved
@zimbatm zimbatm force-pushed the per-build-installer branch 2 times, most recently from 3e50445 to b522f84 Compare November 21, 2020 18:30
@zimbatm
Copy link
Member Author

zimbatm commented Nov 21, 2020

@edolstra is it still good like this? the PR is ready on my side.

The goal is to allow the installation and testing of arbitrary Nix
versions. Extend the base installer to accept a `--tarball-url-prefix
<url>` to change where the Nix tarball is getting downloaded from.

Once this is merged it should allow to:
1. Pick an evaluation at https://hydra.nixos.org/jobset/nix/master that
   looks healthy
2. Select the installedScript build and find the store path.

Now equipped with all of this, use an instance of nar-serve to fetch the
install script and release tarballs:

    curl -sfL https://nar-serve.numtide.com/nix/store/rkv4yh7pym941bhj0849zqdkg2546bdv-installer-script/install \
      | sh --tarball-url-prefix https://nar-serve.numtide.com/nix/store

Or with cachix, strip the /nix/store and derivation name and then:

    curl -sfL https://mycache.cachix.org/serve/rkv4yh7pym941bhj0849zqdkg2546bdv/install \
      | sh --tarball-url-prefix https://mycache.cachix.org/serve

Fixes NixOS#4047
@edolstra edolstra merged commit 56599bd into NixOS:master Nov 21, 2020
@zimbatm zimbatm deleted the per-build-installer branch November 22, 2020 08:39
andersk added a commit to andersk/nix that referenced this pull request May 4, 2021
When commit 233b61d (NixOS#4224) renamed
@binaryTarball_${system}@ to @tarballHash_${system}@, it updated this
reference for every platform except Darwin.arm64.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
system=x86_64-darwin
;;
Darwin.arm64)
hash=@binaryTarball_x86_64-darwin@
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have been updated to @tarballHash_x86_64-darwin@: #4769.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch.

FWIW, Darwin.arm64 was added in #4243, which was opened after this PR and merged shortly before it--I'm pretty sure this was just a slip-up during merge-conflict resolution.

@abathur
Copy link
Member

abathur commented May 4, 2021

I suspect it won't happen since it hasn't already come up, but:

On the off chance anyone tries to backport this to 2.3-maintenance, the fix in #4769 needs to come along for the ride.

zimbatm pushed a commit that referenced this pull request May 11, 2021
When commit 233b61d (#4224) renamed
@binaryTarball_${system}@ to @tarballHash_${system}@, it updated this
reference for every platform except Darwin.arm64.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a working installer script for each commit/PR
7 participants