Macos big sur installer fixes #3996

abathur · 2020-09-08T06:14:17Z

Goal: make sure the installer scripts will run across macOS versions

Caveats, in case this ends up like the last one... 😬

I have tested installs with multiple macOS versions on a single system, but not on Linux.
This work is just focused on procedural differences necessary to install on Big Sur as seen on existing Intel hardware; it has nothing directly to do with the ARM/DTK effort. (I gather the install will fail on a DTK without a release tarball available for that architecture.)

This will also fix #3852 and fix #3957.

A few notes (repeated from a later comment):

I had to fix a few macOS install breaks introduced by other commits already in master. If a new release needs to be cut from master before these changes land, those fixes will need to be cherry-picked/ported to keep macOS installs (for all versions) working correctly.
After daemon installs, commands throw file 'nixpkgs' was not found in the Nix search path... errors on invocation. I don't think I caused this (maybe ec9dd9a did)? If that's correct, it'll need attention outside the scope of this PR.

domenkozar · 2020-09-08T08:14:19Z

cc @LnL7

scripts/create-darwin-volume.sh

LnL7

Looks pretty good to me.

scripts/install-nix-from-closure.sh

scripts/create-darwin-volume.sh

LnL7 · 2020-09-08T21:27:16Z

scripts/create-darwin-volume.sh

-        esac
-        i=$((i+1))
-    done
+    diskutil apfs list -plist "$1" | xmllint --xpath "/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict/key[text()='Name']/following-sibling::string[contains(translate(text(),'N','n'),'nix')]/text()" -


Nice, did you check this with a few conditions?

While thinking through this again I did notice and fix a pair of issues:

after 16d6181 it should break after the first match like the existing loop

after 4b42097 it should match at the beginning of the volume name only

Does that cover what you have in mind?

scripts/create-darwin-volume.sh

abathur · 2020-09-17T22:32:56Z

@domenkozar I think this is roughly ready. Notes:

Despite adding support, 0baa6e7 also adds a temporary check to limit Big Sur installs in case Apple releases it before some issues related to supporting Big Sur in nixpkgs can be ironed out (requested by @LnL7).
Changes here probably merit a broad installer test before the next release
When I tested this on daemon installs, nix-commands throw file 'nixpkgs' was not found in the Nix search path... errors on invocation. I'm planning to open a separate issue for that unless you see something I've done that you think could cause it (I suspect ec9dd9a may be the source).

Atemu · 2020-09-28T12:32:09Z

When I tested this on daemon installs, nix-commands throw file 'nixpkgs' was not found in the Nix search path... errors on invocation.

This already happened on my recent Catalina installation. Doing step 3. from here fixes the issue.

See #2033 for more info.

domenkozar · 2020-09-28T19:57:42Z

@abathur I'm a bit short on time to test this. However, if you generate the installer script, it should be trivial to test with github actions once 11.0 is available for testing.

abathur · 2020-09-28T21:13:00Z

@domenkozar I have them for the last two commits. I'll ping you on IRC.

abathur · 2020-10-08T18:44:26Z

If anyone wants to test this, I've hosted two darwin-x86_64 tarballs with matching stub install scripts (these scripts don't check the tarball hash).

To test the actual Big Sur install, target https://files.t-ravis.com/install-allowed (which will download nix-3.0pre20200914_4b42097-x86_64-darwin.tar.xz, built from 4b42097).
To test the install guards that exclude Big Sur systems until Nixpkgs support is ready, target https://files.t-ravis.com/install-blocked (which will download nix-3.0pre20200917_0baa6e7-x86_64-darwin.tar.xz, built from 0baa6e7).

For reference, I also re-ran some trivial CI jobs with these URLs.

"allowed" run: https://travis-ci.com/github/nix-community/nix-travis-ci/builds/188871929
"blocked" run: https://travis-ci.com/github/nix-community/nix-travis-ci/builds/188873113

Notes:

You shouldn't see any difference between the allowed/blocked installers here; the "blocked" installer should only affect Big Sur+, and travis-ci hasn't added Big Sur runners yet.
The macOS jobs "fail" after successfully installing because there's no nixpkgs pre-specified on the daemon installs (single-user should be OK). I'm not sure if that behavior is correct, but I'm pretty sure it's out-of-scope here.
All of the Linux jobs are canceled--there's no Linux installer.

abathur · 2020-10-15T01:39:35Z

@domenkozar GitHub Actions added Big Sur preview runners today, so I've done comparison runs with the installers above:

Normal run (default installer) with Big Sur in the OS matrix to document existing failure condition https://github.com/abathur/resholve/actions/runs/307640557
Run with "install-allowed" https://github.com/abathur/resholve/actions/runs/307647998
- Installs and tests clean on Catalina
- On Big Sur
  - install-nix-action runs clean
  - but my build does fail with a c compiler issue; this is roughly why @LnL7 asked me to block installs
Run with "install-blocked" https://github.com/abathur/resholve/actions/runs/307655319
- Installs and tests clean on Catalina
- On Big Sur, install fails with the message nixpkgs isn't quite ready to support macOS 11.0 yet

domenkozar · 2020-10-15T11:41:55Z

Could we also run with the tbd branch?

Keeping this commit narrow for reviewability, but some of these conditionals will change in subsequent commits in this PR. Fixes NixOS#3852.

- xpath -> xmllint: xpath's cli interface changed in Big Sur rather than add conditional logic for picking the correct syntax for xpath, I'm changing to xmllint --xpath, which appears to be consistent across versions I've tested... - /plist/dict/key[text()='Writable']/following-sibling::true[1] doesn't do quite what's expected. It was written to try to select a <true /> node paired with the Writable key, but it will also select the *next* <true /> node that appears even if it was paired with another key. - I think there's also a logic bug in the conditionals here. I'm not sure anyone ever actuall saw it, thanks to the xpath bug, though. With the xpath fix, this conditional passes if /nix does not exist, / IS writable, and the version is Catalina+. I think it meant to test for /nix does not exist, / is NOT writable, and the version is Catalina+. I reworked this lightly to make it a little clearer at the code level.

As mentioned in previous commit, Big Sur changes the syntax for the xpath command slightly. In the process of testing out replacements for these, I noticed a few small simplification wins.

Fixes NixOS#3957. Just runs both forms to minimize moving parts.

The move from release.nix to flake.nix appears to have lost some changes from NixOS#3628 / 1c56f18, leaving create-darwin-volume.sh out of the release tarball. Under the assumption that this was just an accident/byproduct of when flake.nix split off and not intentional, I am restoring those edits.

Some of the changes in NixOS#3788 to support non-systemd Nix installs don't appear to be aware that the darwin installer exists, which resulted in some skipped steps and inappropriate instructions.

Env vars for ZSH were moved from /etc/zshrc to /etc/zshenv in NixOS#3608 to address an issue with zshrc getting clobbered by OS updates, but /etc/zshenv doesn't exist by default--so *nothing* would get set up for zsh users unless they already happened to have /etc/zshenv. Creating these files if they don't exist. Also cut separate creation of profile.d/nix.sh, which isn't needed now.

abathur · 2020-10-19T19:17:28Z

I've force-pushed a squashed history which collapses all of the incremental fix commits back down into:

4 commits fixing issues directly necessary for Big Sur, which can be cherry-picked to 2.3-maintenance: 9c3dc9d 1f02b65 e736f8f fe80790
3 commits fixing issues problems already present in master: 3a8699a b719f68 c40bad4
1 commit which blocks install on Big Sur install, which we can eventually drop/revert depending on the status of the tbd/stdenv work: f289bdb

domenkozar · 2020-10-20T10:19:00Z

Thanks!

domenkozar · 2020-10-20T10:22:23Z

Cherry-picked 4 commits to 2.3-maintenance, we'll hopefully release 2.3.8 once nixpkgs is ready.

abathur · 2020-10-25T21:26:20Z

@domenkozar I didn't notice that 2.3.8 has already been cut or I would've done this sooner, but I ran an install test just now on my laptop with Big Sur:

 - system: `"x86_64-darwin"`
 - host os: `Darwin 20.1.0, macOS 10.16`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.8`
 - channels(abathur): `"nixpkgs-21.03pre246624.cfed29bfcb2"`
 - nixpkgs: `/Users/abathur/.nix-defexpr/channels/nixpkgs`

domenkozar · 2020-10-25T21:31:03Z

Nice!!

matthewbauer reviewed Sep 8, 2020

View reviewed changes

scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved

LnL7 reviewed Sep 8, 2020

View reviewed changes

cole-h reviewed Sep 11, 2020

View reviewed changes

scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved

abathur force-pushed the macos_big_sur_fixes branch 2 times, most recently from 7ccb5b0 to af6a895 Compare September 13, 2020 03:27

abathur marked this pull request as ready for review September 14, 2020 15:49

abathur mentioned this pull request Oct 8, 2020

Create a working installer script for each commit/PR #4047

Closed

abathur changed the title ~~Macos big sur fixes~~ Macos big sur installer fixes Oct 15, 2020

abathur added 8 commits October 19, 2020 12:22

update macOS version handling for Big Sur

9c3dc9d

Keeping this commit narrow for reviewability, but some of these conditionals will change in subsequent commits in this PR. Fixes NixOS#3852.

replace xpath with xmllint --xpath; simplify

e736f8f

As mentioned in previous commit, Big Sur changes the syntax for the xpath command slightly. In the process of testing out replacements for these, I noticed a few small simplification wins.

adapt to apfs.util flag diff in catalina/big sur

fe80790

Fixes NixOS#3957. Just runs both forms to minimize moving parts.

fix skipped multi-user install steps on macOS

b719f68

Some of the changes in NixOS#3788 to support non-systemd Nix installs don't appear to be aware that the darwin installer exists, which resulted in some skipped steps and inappropriate instructions.

discourage casual Big Sur installs

f289bdb

abathur force-pushed the macos_big_sur_fixes branch from 0baa6e7 to f289bdb Compare October 19, 2020 17:34

domenkozar merged commit e0ca98c into NixOS:master Oct 20, 2020

stephank mentioned this pull request Nov 3, 2020

Fix missing create-darwin-volume.sh in installer #4074

Closed

abathur deleted the macos_big_sur_fixes branch March 4, 2021 19:03

ldesgoui mentioned this pull request Aug 1, 2021

Write Nix 2.4 release notes #4521

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Macos big sur installer fixes #3996

Macos big sur installer fixes #3996

abathur commented Sep 8, 2020 •

edited

domenkozar commented Sep 8, 2020

LnL7 left a comment

LnL7 Sep 8, 2020

abathur Sep 14, 2020

abathur commented Sep 17, 2020

Atemu commented Sep 28, 2020

domenkozar commented Sep 28, 2020

abathur commented Sep 28, 2020

abathur commented Oct 8, 2020

abathur commented Oct 15, 2020 •

edited

domenkozar commented Oct 15, 2020

abathur commented Oct 19, 2020

domenkozar commented Oct 20, 2020

domenkozar commented Oct 20, 2020

abathur commented Oct 25, 2020

domenkozar commented Oct 25, 2020

Macos big sur installer fixes #3996

Macos big sur installer fixes #3996

Conversation

abathur commented Sep 8, 2020 • edited

domenkozar commented Sep 8, 2020

LnL7 left a comment

Choose a reason for hiding this comment

LnL7 Sep 8, 2020

Choose a reason for hiding this comment

abathur Sep 14, 2020

Choose a reason for hiding this comment

abathur commented Sep 17, 2020

Atemu commented Sep 28, 2020

domenkozar commented Sep 28, 2020

abathur commented Sep 28, 2020

abathur commented Oct 8, 2020

abathur commented Oct 15, 2020 • edited

domenkozar commented Oct 15, 2020

abathur commented Oct 19, 2020

domenkozar commented Oct 20, 2020

domenkozar commented Oct 20, 2020

abathur commented Oct 25, 2020

domenkozar commented Oct 25, 2020

abathur commented Sep 8, 2020 •

edited

abathur commented Oct 15, 2020 •

edited