darwin: encrypt nix volume if filevault is enabled #4289

abathur · 2020-11-28T04:42:20Z

This refactors WIP from #4181 to address things that became obvious:

The macOS install is getting too complex for single-user mode.
There's already frustration around difficulty of uninstalling multi-user on macOS.
I want to support upgrading existing macOS volumes to work the same as those made by this WIP, but simply tacking it on as an extra option at all relevant points was looking like a lot of extra conditional logic and calls to some fairly slow diskutils commands.
The greater number of Nix-related artifacts shook loose additional concern around avoiding state/artifact inconsistencies.
As I developed a curing process to address these inconsistencies, I realized it won't take too much more work to extend these concepts to uninstalling the other Darwin and general Nix multi-user artifacts. I'm pleasantly surprised how practical an uninstall mode feels from here.

The history had grown unruly even before this refactor, so for now I'm flattening this into an omnibus commit to start fresh. Since this touches on a lot, I've tried to better-document the high-altitude view by making three outlines:

~~the general process/logic in pseudocode (as-is, and for the uninstaller I'm imagining)~~ removed this for merge
how high-level user-facing functionality changes here
what lower-level implementation changes I'm making

High-level functionality changes

disable single-user installs on macOS
if the user's root is read-only, create a volume (overridable default)
- --darwin-use-unencrypted-nix-store-volume flag is now a no-op warning
- if the boot volume has FileVault enabled, encrypt this volume
  - generate a random credential and place it in the system keychain
- create a LaunchDaemon to mount it
  - still use fstab to specify mount opts (rw,noauto,nobrowse,suid,owners)
    - noauto so that the system won't auto-mount if our mounter breaks (goal: the user finds/presents for help)
  - the daemon isn't strictly necessary for unencrypted volumes, but I'm using it in both cases for consistency
update volume naming/customization
- default: use or create volume named "Nix Store" on same disk as /
  - I had changed this to "Nix", but I want to support up-converting current "Nix Store" volumes and can't justify the complexity of supporting both names.
  - no longer tries to match and use volumes labeled [nN]ix.* (see [OSX catalina] install script fails if a volume starting with [nN]ix.* already exists #4150)
- overridable variables for the filesystem, label, and target disk
- overridable variables to BYOV: volume special device, and UUID
support curing any volume-related state (if present) on install
- if Nix Store is encrypted
  - verify we can unlock it from keychain
  - otherwise, prompt the user to delete it or manually add the pw to keychain
- if filevault is on but Nix Store isn't encrypted, offer to encrypt it
- if nix is the only item in fstab and synthetic.conf, remove them; otherwise suggest edits that will remove the nix line
- remove the mounter launchdaemon

Lower-level implementation changes

scaffold uninstall and combined uninstall+install actions
multi-user installer
- add reminder/remind UI/X idiom (add notes as we go, show all at end)
  - no more saving variables or re-testing conditions at the end
  - if interactive: show one at a time and require acknowledgement
  - convert some existing messages to use this
- add password_confirm UI/X idiom for actions like deleting a volume
  - forces a sudo prompt (even when headless or sudo was recently authed)
- add confirm_rm and confirm_edit UI/X idioms
  - these are "for" curing volume state, but I anticipate them being useful for uninstaller (and, I guess, for the installer if we wanted to be more interactive/explicit around some edits)
- try to make create_directories more idempotent (may cause trouble?)
  - if /nix already exists, chown -R root:nixbld it
  - use install -dv -m instead of mkdir -pv -m
  - combine dir-structure steps 2 & 3 with install's -g (group) flag
- only defer to a pre-set NIX_SSL_CERT_FILE if its physical path is not in /nix/store
  - multi-user reinstall over a single-user install was skipping cert setup
- fail earlier (before sudo lecture etc.) if EUID=0
- add two new install phases:
  - (poly_)cure_artifacts for reverting changes Nix makes
  - poly_prepare_to_install for preparing the Nix volume on macOS
- normalize SCRATCH/mktemp path when macOS /usr/bin/mktemp is used:
  - /usr/bin/mktemp -d -t tmp.XXXXXXXXXX --> /.../tmp.XXXXXXXXXX.2RMrVwkx
  - /usr/bin/mktemp -d "${TMPDIR:-/tmp/}tmp.XXXXXXXXXX" --> /.../tmp.JwmZDxN1kf
- _sudo "explanation" command ... wrapper now sends explanation to stderr
  - make it easier to pipe commands using the wrapper; see create-darwin-volume.sh:volume_pass_works() for an example
refactor create-darwin-volume.sh to
- be sourceable as a library but still support direct invocation
  - use UI/X idioms from the multi-user installer
  - give multi-user installer fine-grained control of what runs when
  - ~legacy (bare) direct invocation
    - in case we have to reinstate single-user
    - avoid no/low-notice breaks for anyone who has been directly running it
  - externally invoke a non-main func with args "no-main" "funcname" ...
  - I've seen a fair number of modified forks of this script; over time I hope sourceability and external function invocation will make it easier to re-use
- largely eradicate a pattern, <run very slow diskutil command> | <parse a single bit of info out with an xpath query>, which is OK in isolation but was becoming a bit too common
- remove (most) instruction dumps that describe how to remove things that the curing process can now handle
- defensively use absolute default paths for some common executables (coreutils, find, grep, etc.)

@LnL7 @lilyball @matthewbauer @dhess @mroi @domenkozar (others I'm forgetting are inevitably also interested; feel free to add people)

Try it out

I have been hosting my own installers for much of the history of this PR. Nix now has a better installer-testing process, so I'm going to see if we can lean on it here. I'll leave the old process note here for a bit in case there are problems with the new approach. If you can't try it out, you can see output from the CI run.

The new approach requires an extra flag, so it'll be easiest to just use in interactive mode:

sh <(curl https://abathur-nix-install-tests.cachix.org/serve/yihf8zbs0jwph2rs9qfh80dnilijxdi2/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve

Here's a history of the versions it has pointed at, with the current at the bottom (note: I'll only update this for substantive changes or breakage):

~~6132132~~ (equivalent to 401afa0, but without the reference darwin_volume_reference.py)
~~6bb3bc9~~
~~0bd1083~~
~~86dbee7~~
~~97269f6~~
~~c235bed~~
eab14a6

old privately-hosted installer (out of date--use only if above doesn't work)

I have a temporary installer up at https://files.t-ravis.com/install-multi, so curl https://files.t-ravis.com/install-multi | sh (headless) and sh <(curl https://files.t-ravis.com/install-multi) (interactive) should work.

Other issues

I believe this PR:

dhess · 2020-11-28T11:00:22Z

Thanks once again for this work and the attention to detail.

Does anyone know: on T2 Macs, if you encrypt a user-created APFS volume, is the encryption/decryption for that volume still done on the T2, as with the system volumes; or is it done in software on the CPU?

abathur · 2020-11-28T16:20:03Z

on T2 Macs, if you encrypt a user-created APFS volume, is the encryption/decryption for that volume still done on the T2, as with the system volumes; or is it done in software on the CPU?

I think it's on the T2, but I won't swear I personally know this as a fact. For example, see the last line below:

$ diskutil apfs encryptVolume disk1s7 -user disk -passphrase foo
Encrypting with the new "Disk" crypto user on disk1s7
The new "Disk" user will be the only one who has initial access to disk1s7
The new APFS crypto user UUID will be 920D4EAD-BFAE-457B-9A19-827AAC0C1B42
Encryption has likely completed due to AES hardware; see "diskutil apfs list"

If you encrypt a volume on a pre-T2 mac, that line will say Background encryption is ongoing; see "diskutil apfs list" to see progress

LnL7 · 2020-11-28T17:30:40Z

That's also how I understood the T2 documentation from apple https://www.apple.com/jp/mac/docs/Apple_T2_Security_Chip_Overview.pdf.

The chip handles all the encryption and is actually always enabled, FileVault only changes the conditions for which the chip unlocks the data. Meaning that the data also doesn't have to change when updating credentials. Which the behaviour you noticed seems to confirm is the case.

scripts/install-multi-user.sh

abathur · 2020-11-28T21:41:22Z

In case anyone's curious, I added screenshots of the ~interactive UI/X elements in a comment alongside the code for each (and immediately resolved them, so they don't take up gobs of space in the main thread).

lilyball · 2020-12-01T00:06:16Z

For future reference, you can use <details> tags to hide screenshots by default.

<details><summary>Description of screenshot</summary>

// drag image into this blank line, if it's surrounded by blank lines the markdown ![image](url) syntax will work just fine.

</details>

This will show up as

Collapsed summary here

Once expanded the image will be visible.

lilyball

This is only a partial review, I’ll have to come back to this later.

doc/manual/src/installation/installing-binary.md

scripts/create-darwin-volume.sh

scripts/install-multi-user.sh

scripts/install-systemd-multi-user.sh

scripts/install-multi-user.sh

abathur · 2020-12-06T03:46:44Z

I've pushed commits addressing one round of comments from @lilyball and @SuperSandro2000 (also thanks to @samueldr and @jtojnar for additional discussion). Going to go ahead and convert this from a draft.

I've started a build of this locally, and will update to confirm its status (status: my install checks are working as of 6bb3bc9, but the Nix repo checks failed at this commit--I'd be surprised if this caused that, so I assume it's a transient failure or flakiness?)

For anyone champing at the bit, I suspect we'll be holding up just a little bit (if it's a short wait) for the chance I'll be able to help @domenkozar test a new GitHub Actions integration per #4047 that should make it easier to test PRs that touch the installer.

In any case, reviews still welcome in the meantime. :)

lilyball

Another partial review. Apologies for not managing to do a full one.

scripts/create-darwin-volume.sh

lilyball · 2020-12-18T00:35:06Z

scripts/create-darwin-volume.sh

@@ -1,33 +1,235 @@
-#!/bin/sh
-set -e
+#!/usr/bin/env bash


I'm not sure if this really matters, but maybe we should stick with /bin/bash just to ensure we're using the exact same version of bash everywhere?

Good question.

I just adopted the shebang from install-multi-user.sh for consistency.

I'm not sure how to weight consistency with install-multi-user against being explicit here. Everything install-multi-user.sh sources has to be 3.2 compatible for this reason (and I did a lot of manual 3.2 testing across all of the scripts as I went).

I assumed/inferred from the #!/usr/bin/env bash shebang on install-multi-user that at least some of the platforms we might be run on don't have a /bin/bash, but if that's not true I suppose they could all change.

What do you think?

I suppose consistency is worthwhile. If we do need to run install-multi-user.sh on a platform that doesn't have /bin/bash then the current shebang makes sense.

scripts/create-darwin-volume.sh

abathur · 2020-12-20T01:55:57Z

bf5290d fixes an oversight caught by lilyball regarding curing/checking for the presence of Nix as a symlink in synthetic.conf. You can see sample output in this CI run:

The stitched jobs fail because once we've stitched, we need a reboot to remove the symlink.

ethnt · 2021-01-02T20:03:52Z

@abathur Thanks a lot for your work on this! Is there an ETA for when this will be made publicly available?

abathur · 2021-01-02T22:16:21Z

Not sure. We're on the crowded side of a bottleneck. :)

grahamc · 2021-04-29T19:59:06Z

I've seen good success with this PR and I think it is time to give our users a better experience, so I'm going to go ahead and merge it :)

jgeerds · 2021-04-30T21:48:36Z

@abathur Thank you very much for your work! Am I right that these changes only affect nixUnstable and upcoming releases? If so, is there a chance for a 2.3.x backport?

abathur · 2021-04-30T22:26:22Z

@jgeerds Yep.

Backport decisions are above my paygrade, but unless 2.4 is impending I think it spares enough pain to be worth a backport and release.

abathur · 2021-05-01T03:11:45Z

This may also be a good place to note: I think the merge of this PR lights a path to full uninstall/reinstall. I can't pursue it any time soon, but I'm happy to discuss as needed if someone else can. I wrote a proof-of-concept in Oct. that should be reheatable. Validation is also much easier since domen added installer tests.

nixos-discourse · 2021-05-03T12:54:48Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/2

rpearce · 2021-05-13T02:05:33Z

Holy smokes; what a great install experience. This worked for me on a macOS early 2013 running 10.15.7. Thank you!!!

rpearce · 2021-05-18T02:19:15Z

@abathur Sorry to bother: how will we know when this will become the default installer for Darwin? This is what I'm telling coworkers currently:

sh <(curl https://abathur-nix-install-tests.cachix.org/serve/yihf8zbs0jwph2rs9qfh80dnilijxdi2/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve

abathur · 2021-05-18T03:26:42Z

@rpearce No worries. Subscribing to this thread is probably the best bet.

I deferred a sprawling, intricate refactor of another project for a few months to focus on this PR. For my own sanity, I had to stick a fork in this a couple months back so that I could shift enough of my focus back to that effort to make any headway.

I guess we're waiting for either:

a new release from master (whether that's 2.4 or 3.0)
someone to drive an effort to backport it to the 2.3-maintenance branch and get a new release from there

It can be backported (and I agree with releasing it), but there are a few caveats that probably make it more complicated than just bugging domen:

There are other commits in master that haven't been backported, and this PR is extensive enough that I imagine someone would need to audit the other changes there to make sure anything this builds on (and any fixes to this?) get pulled in. A good place to start is probably something like: git log --color=always --format='%C(auto,blue)%h%Creset %C(auto,green)%cs%Creset [%C(auto,red)%<|(37)%S%Creset]%C(auto,yellow) %m %Creset%s' --cherry-mark --no-merges master...2.3-maintenance -- scripts | grep -v ' = '
The documentation format was converted from xml to markdown between 2.3-maintenance and this work, so the doc changes will need to be re-applied in the xml.
The installer-testing work domen implemented this winter isn't present in 2.3-maintenance, so someone will need to either backport that work as well, or manually generate installers to test them (i.e., the old way :/)

nixos-discourse · 2021-05-18T18:21:02Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/anyone-up-for-picking-at-some-nix-onboarding-improvements/13152/1

nixos-discourse · 2021-06-25T22:38:28Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-install-nix-on-macos-with-m1/13791/4

mpscholten · 2021-07-06T05:20:40Z

An issue was just reported to IHP with the following error running this installer:

SHA-256 hash mismatch in 'https://abathur-nix-install-tests.cachix.org/serve/jqr52m6w8dwag5932hw67kq0rvbnji01/nix-2.4pre19700101_eab14a6-x86_64-darwin.tar.xz'; expected @binaryTarball_x86_64-darwin@, got eb40f032d84e2fadb1e875ad7a35a075f9c5b47a4737084a87f925292378b57a

Was anything changed on the installer? Did it already get released and @abathur's installer is deprecated?

Linked issue: digitallyinduced/ihp#960

domenkozar · 2021-07-06T11:19:40Z

Most likely a transient network error, the hash is correct when I test it locally.

mpscholten · 2021-07-06T12:46:09Z

Thanks for the quick help Domen! :)

The user reported back that the problem still occurs. Notable it's happening on a Macbook Air with a M1. Maybe this could be causing the problems?

domenkozar · 2021-07-06T13:36:52Z

The installer that's used by IHP doesn't support M1, I'd suggest to use the newest one from Nix master or use the default installer that ships M1 support as of yesterday.

mpscholten · 2021-07-06T17:29:38Z

Thanks, using the default installer solved the problem :)

nixos-discourse · 2021-09-21T19:27:42Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-is-the-recommended-installer-to-use-macos/15138/1

abathur mentioned this pull request Nov 28, 2020

darwin: encrypt nix volume if filevault is enabled #4181

Closed

abathur commented Nov 28, 2020

View reviewed changes

scripts/install-multi-user.sh Show resolved Hide resolved

abathur commented Nov 28, 2020

View reviewed changes

scripts/install-multi-user.sh Show resolved Hide resolved

abathur commented Nov 28, 2020

View reviewed changes

scripts/install-multi-user.sh Show resolved Hide resolved

abathur commented Nov 28, 2020

View reviewed changes

scripts/install-multi-user.sh Show resolved Hide resolved

lilyball reviewed Dec 1, 2020

View reviewed changes

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

scripts/install-multi-user.sh Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

scripts/install-multi-user.sh Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

scripts/install-systemd-multi-user.sh Show resolved Hide resolved

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

scripts/install-multi-user.sh Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

scripts/install-multi-user.sh Show resolved Hide resolved

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

scripts/install-multi-user.sh Show resolved Hide resolved

abathur marked this pull request as ready for review December 6, 2020 03:34

lilyball mentioned this pull request Dec 16, 2020

Multi-user Nix install doesn't initialize for non-interactive login bash shell on macOS #4376

Open

lilyball reviewed Dec 19, 2020

View reviewed changes

abathur force-pushed the encrypt_darwin_volume branch from ca42caf to bf5290d Compare December 20, 2020 00:19

abathur mentioned this pull request Dec 23, 2020

Reinstalling on macos: "/nix: Read-only file system" #4390

Closed

grahamc merged commit d15a196 into NixOS:master Apr 29, 2021

mpscholten mentioned this pull request Apr 30, 2021

Error installing Nix in Mac with FileVaut encrypted volumes digitallyinduced/ihp#93

Closed

chkoss mentioned this pull request May 6, 2021

doc: fix broken link #4752

Closed

abathur mentioned this pull request May 13, 2021

Multi user Installation failed #4797

Closed

ericbmerritt mentioned this pull request May 24, 2021

Non-standard store location support cachix/cachix#301

Open

abathur mentioned this pull request Jun 2, 2021

Support for Apple Silicon (aarch64-darwin) NixOS/nixpkgs#95903

Closed

abathur mentioned this pull request Jun 11, 2021

nix-daemon doesn't have access to nix binaries in non-NixOS multi-user installation #4902

Open

abathur mentioned this pull request Jul 3, 2021

Add option to allow users to inspect the system-specific install script in $tempDir after failed installations #4979

Open

abathur mentioned this pull request Jul 6, 2021

Manual: Note in macOS installation section is incomplete #4987

Closed

abathur mentioned this pull request Aug 1, 2021

Inconsistent installation instructions #5080

Open

ldesgoui mentioned this pull request Aug 1, 2021

Write Nix 2.4 release notes #4521

Closed

lilyinstarlight mentioned this pull request Aug 30, 2021

Install fails on Catalina nix-community/nix-unstable-installer#13

Closed

abathur mentioned this pull request Sep 11, 2021

Macos M1 reinstall failure on LaunchDaemon setup #5236

Open

abathur mentioned this pull request Feb 24, 2022

doc: Add detailed uninstall section for macOS #6144

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

darwin: encrypt nix volume if filevault is enabled #4289

darwin: encrypt nix volume if filevault is enabled #4289

abathur commented Nov 28, 2020 •

edited

dhess commented Nov 28, 2020

abathur commented Nov 28, 2020

LnL7 commented Nov 28, 2020

abathur commented Nov 28, 2020

lilyball commented Dec 1, 2020

lilyball left a comment

abathur commented Dec 6, 2020 •

edited

lilyball left a comment

lilyball Dec 18, 2020

abathur Dec 19, 2020

lilyball Jan 9, 2021

abathur commented Dec 20, 2020

ethnt commented Jan 2, 2021

abathur commented Jan 2, 2021 •

edited

grahamc commented Apr 29, 2021

jgeerds commented Apr 30, 2021

abathur commented Apr 30, 2021

abathur commented May 1, 2021

nixos-discourse commented May 3, 2021

rpearce commented May 13, 2021

rpearce commented May 18, 2021

abathur commented May 18, 2021 •

edited

nixos-discourse commented May 18, 2021

nixos-discourse commented Jun 25, 2021

mpscholten commented Jul 6, 2021

domenkozar commented Jul 6, 2021

mpscholten commented Jul 6, 2021

domenkozar commented Jul 6, 2021

mpscholten commented Jul 6, 2021

nixos-discourse commented Sep 21, 2021

darwin: encrypt nix volume if filevault is enabled #4289

darwin: encrypt nix volume if filevault is enabled #4289

Conversation

abathur commented Nov 28, 2020 • edited

High-level functionality changes

Lower-level implementation changes

Try it out

Other issues

dhess commented Nov 28, 2020

abathur commented Nov 28, 2020

LnL7 commented Nov 28, 2020

abathur commented Nov 28, 2020

lilyball commented Dec 1, 2020

lilyball left a comment

Choose a reason for hiding this comment

abathur commented Dec 6, 2020 • edited

lilyball left a comment

Choose a reason for hiding this comment

lilyball Dec 18, 2020

Choose a reason for hiding this comment

abathur Dec 19, 2020

Choose a reason for hiding this comment

lilyball Jan 9, 2021

Choose a reason for hiding this comment

abathur commented Dec 20, 2020

ethnt commented Jan 2, 2021

abathur commented Jan 2, 2021 • edited

grahamc commented Apr 29, 2021

jgeerds commented Apr 30, 2021

abathur commented Apr 30, 2021

abathur commented May 1, 2021

nixos-discourse commented May 3, 2021

rpearce commented May 13, 2021

rpearce commented May 18, 2021

abathur commented May 18, 2021 • edited

nixos-discourse commented May 18, 2021

nixos-discourse commented Jun 25, 2021

mpscholten commented Jul 6, 2021

domenkozar commented Jul 6, 2021

mpscholten commented Jul 6, 2021

domenkozar commented Jul 6, 2021

mpscholten commented Jul 6, 2021

nixos-discourse commented Sep 21, 2021

abathur commented Nov 28, 2020 •

edited

abathur commented Dec 6, 2020 •

edited

abathur commented Jan 2, 2021 •

edited

abathur commented May 18, 2021 •

edited