[WIP] nixos/systemd-boot: boot counting and automatic fallback #84204

danielfullmer · 2020-04-03T18:49:35Z

Motivation for this change

This draft PR adds support for systemd-boot "automatic boot assessment".
For background information, please first read the following document: https://systemd.io/AUTOMATIC_BOOT_ASSESSMENT/

Setting boot.loader.systemd-boot.counters.enable = true enables this optional functionality.
Since what a "successful boot" exactly means is somewhat context dependent, systemd provides boot-complete.target which serves as a synchronization point that an end user could order necessary services before. The boot entry will be marked as successfully booting only if this target succeeds.
For unattended boots, there are some other potentially useful options such as adding panic=1 to boot.kernelParams or setting FailureAction=reboot for certain necessary services. However, these are end-user concerns that are somewhat orthogonal to boot counting and automatic fallback, and they probably do not belong in this PR.

This is a draft PR, as there are a few issues that still need to be resolved:

The menu entries appear to be ordered incorrectly upstream: systemd-boot menu ordering and boot counting systemd/systemd#15256 (fixed upstream, should be in v246)
The systemd-boot test needs a writable boot disk (since boot counters are state included in the entry filenames). I have a hack enabling this, which should be cleaned up somehow.
Documentation is needed

(edit: the issue below has been resolved to my satisfaction)

Perhaps the biggest current issue which could use some feedback on is how we set the "default" for systemd-boot. Normally, systemd-boot-builder.py sets the default entry in loader.conf. The logic in systemd-boot means that this entry would always be selected--overriding anything having to do with boot counters. I've disabled setting default in this draft PR, and so systemd-boot should select the latest generation which does not have 0 tries left. However, this would break the ability for the user to switch back to a previous generation, which is clearly undesirable. Perhaps we need to further patch systemd-boot to use default only if it has remaining tries left, and otherwise select the latest generation with remaining tries left.

Things done

grahamc · 2020-04-11T21:24:09Z

cc @adisbladis for interest

flokli · 2020-04-23T07:48:32Z

@danielfullmer with 245.3 merged into master, can you rebase this PR and see if it's still broken upstream?

danielfullmer · 2020-04-23T17:57:09Z

@flokli Sure, just rebased. There haven't been any relavent changes to systemd-boot in the meantime.

Since I haven't received any response about the menu ordering here systemd/systemd#15256, I'll send a patch upstream to see if that sparks any discussion.

I also recently patched systemd-boot to ignore default entries which have no tries left. That allows me to remove the hack where I commented out the default setting in systemd-boot-builder.py. I'll see if I can get any upstream feedback on whether the original behavior was intended or not.

Just to be clear, the logic is now this: We select the default entry specified in loader.conf (which should be the current generation), unless this entry has zero tries left. In that case, we select the entry with the highest id (entry filename--basically, generation number) which has a nonzero number of tries left.

Upstream PR here: systemd/systemd#15577

flokli · 2020-06-14T12:44:12Z

I just prepared the bump to 245.6, which doesn't seem to include systemd/systemd#15577 - do we need to wait for 9eaac58683788ae1870760ad61d594c8ce7a43a8 to have landed in our systemd package?

We also might need to address #86422 (comment), so actual sd-boot fixes manifest on existing installations.

danielfullmer · 2020-06-14T14:44:20Z

@flokli Yeah, it's probably best to wait for the next major systemd release before merging.

I actually have a little more free time now. So I'll see if I can take a look at #86422 again either today or tomorrow.

danielfullmer · 2020-06-16T06:07:57Z

I've resolved the earlier issue with the default selection to my satisfaction. To review, the problem is that systemd will always select an entry matching default set in loader.conf, even if that entry didn't have any tries left. Since we currently set default to only match a single entry, it would never fallback to any other entry.

My fix was this: instead of setting (for example) default nixos-generation-15, we now set default nixos-generation-*, and rely on the systemd-boot's lexicographic ordering of the entry filenames to ensure we actually default to the proper entry. This way, if the default boot entry has no tries left, there could still be another entry matching nixos-generation-* which systemd-boot could try instead.

Since the proper default generation might not be the highest numbered generation, I've renamed the default entry to (for example) nixos-generation-default-15. This would be compared against other entries named (for example)nixos-generation-20. Since 'd' is lexicographically greater than '1' (or any other digit), nixos-generation-default-15 is ordered last, which is the one that systemd-boot chooses as the default selection. This does seem a little hackish, but I think this is the best approach. I will add comments to the source explaining this.

This approach has the additional benefit that non-default system profiles would be respected. If the user has run nixos-rebuild boot --profile-name test, and the latest test generation is not booting correctly, systemd-boot should only choose other test generations, not any other profile's generations.

cole-h · 2021-03-22T18:43:51Z

Hi @danielfullmer,

I just wanted to swing by and say: thank you for your work on this! I tested this on my machine just now and it works flawlessly.

Is there anything I can do to help move this into a mergeable state?

EDIT: Removed from the Staging project because it looks like this is no longer a mass rebuild.

cole-h

Something I just noticed (after having switched off of using this PR) -- I still have boot entries with counters attached, but I also have the same profiles without counters. e.g.:

$ ls /boot/loader/entries
[...]
.rwxr-xr-x 690 root 22 Mar 11:49 nixos-generation-70+0-1.conf
.rwxr-xr-x 580 root 23 Mar 13:29 nixos-generation-70.conf
.rwxr-xr-x 690 root 22 Mar 11:49 nixos-generation-71+0-1.conf
.rwxr-xr-x 580 root 23 Mar 13:29 nixos-generation-71.conf
.rwxr-xr-x 690 root 22 Mar 11:49 nixos-generation-72+0-1.conf
.rwxr-xr-x 580 root 23 Mar 13:29 nixos-generation-72.conf
.rwxr-xr-x 690 root 22 Mar 11:49 nixos-generation-73+0-3.conf
.rwxr-xr-x 580 root 23 Mar 13:29 nixos-generation-73.conf
.rwxr-xr-x 580 root 23 Mar 13:29 nixos-generation-74.conf
.rwxr-xr-x 580 root 23 Mar 13:29 nixos-generation-76.conf
.rwxr-xr-x 690 root 22 Mar 11:49 nixos-generation-default-74+3.conf

If boot counters are ever disabled and we see that there are profiles with a counter attached, what should be done? Nothing (and continue to regenerate the counted profiles on nixos-rebuild switch)? Skip it (e.g. don't regenerate that profile)?

cole-h · 2021-03-23T20:47:11Z

nixos/modules/system/boot/loader/systemd-boot/systemd-boot-builder.py

+
+        generation_dir = os.readlink(system_dir(self.profile, self.generation))
+        tmp_path = self.path + ".tmp"
+        kernel_params = "systemConfig=%s init=%s/init " % (generation_dir, generation_dir)


As of 5276ebb, systemConfig has been removed.

Suggested change

kernel_params = "systemConfig=%s init=%s/init " % (generation_dir, generation_dir)

kernel_params = "init=%s/init " % generation_dir

stale · 2021-09-21T19:24:48Z

I marked this as stale due to inactivity. → More info

RaitoBezarius · 2021-10-17T22:13:00Z

This still looks interesting, can we do something to help?

stale · 2022-04-16T03:15:37Z

I marked this as stale due to inactivity. → More info

RaitoBezarius · 2022-04-16T13:49:32Z

Still interested, can we help you @danielfullmer ?

danielfullmer · 2022-04-16T21:55:54Z

Man this has been on my TODO list for so long, apologies for not taking it the last 20% and trying to get it merged. I now have a concrete need for this feature so I intend to work on this soon (on the order of weeks). If anyone gets to it before I do: I know we'll at least need to (1) Resolve any conflicts and ensure the feature still works (2) Figure out a solution to @cole-h 's issue of how to handle boot entries if we turn the feature off and on, and (3) Write some documentation/release notes for the feature. This is very likely too late to make it into 22.05, but it'd be nice to get it merged into master for the release after that.

lovesegfault · 2022-07-26T04:52:32Z

@danielfullmer Gentlest of pings :)

dasJ · 2022-07-26T12:32:31Z

Partly related, I have a half-baked implementation of this for grub at home. Main issue is that it's shitty to implement when we want proper counting and that I cannot get the test runner to do what I want

domenkozar · 2022-08-08T09:35:12Z

As part of Cachix Deploy, I'd very much like to sponsor this work to get it fixed for everyone in NixOS itself. Let me know if someone is willing to get it done :)

danielfullmer · 2022-08-09T04:13:47Z

@domenkozar I'm actually currently working on bringing this PR up-to-date with master right now. I know I've been meaning to work on this for a while now, and I don't want to prevent you from sponsoring someone else to work on this if needed. But if I don't have something within ~2 weeks, then take that as a sign that I might not get to it for a while longer.

samueldr · 2022-08-09T16:59:45Z

@domenkozar I'm actually currently working on bringing this PR up-to-date with master right now. I know I've been meaning to work on this for a while now, and I don't want to prevent you from sponsoring someone else to work on this if needed. But if I don't have something within ~2 weeks, then take that as a sign that I might not get to it for a while longer.

Are you unable to get sponsored for this? Not sure if that's what @domenkozar could have had in mind.

danielfullmer · 2022-08-10T01:26:57Z

Are you unable to get sponsored for this? Not sure if that's what @domenkozar could have had in mind.

It might be complicated for some unrelated reasons--but I'll reach out directly.

RaitoBezarius · 2023-06-03T22:45:33Z

As part of Cachix Deploy, I'd very much like to sponsor this work to get it fixed for everyone in NixOS itself. Let me know if someone is willing to get it done :)

I'm interested into taking over for this and bringing this to completion if @danielfullmer cannot really anymore.

nixos-discourse · 2023-08-08T00:15:37Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-did-you-get-stuck-in-the-nix-ecosystem-tell-me-your-story/31415/6

nixos-discourse · 2023-12-03T14:01:53Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pi4-broken-on-23-11-uefi-zfs-root/36375/3

n8henrie · 2023-12-08T16:00:55Z

@dasJ Could you share more about your grub implementation?

Would love to see this PR go forward!

RaitoBezarius · 2023-12-09T12:11:45Z

Closing in favor of #273062 which is close to completion.

ofborg bot added 6.topic: nixos 8.has: module (update) labels Apr 3, 2020

ofborg bot requested review from Mic92, edolstra, andir and flokli April 3, 2020 18:56

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Apr 3, 2020

FRidh added this to WIP in Staging Apr 3, 2020

danielfullmer mentioned this pull request Apr 11, 2020

nixos/systemd-boot: Add basic test #85029

Merged

10 tasks

deliciouslytyped mentioned this pull request Apr 15, 2020

Request for fail-safe rollback feature #65477

Open

danielfullmer force-pushed the systemd-boot-assessment branch 2 times, most recently from 9584565 to 4c5d3e4 Compare April 16, 2020 01:05

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 and removed 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Apr 16, 2020

danielfullmer force-pushed the systemd-boot-assessment branch from 4c5d3e4 to 2fe51f7 Compare April 23, 2020 16:57

danielfullmer mentioned this pull request Apr 23, 2020

sd-boot: fix menu ordering with boot counting systemd/systemd#15577

Merged

lovesegfault mentioned this pull request May 27, 2020

Support Automatic Reboots infinisil/nixus#3

Open

danielfullmer force-pushed the systemd-boot-assessment branch from 0db9bed to 2f2e083 Compare June 14, 2020 03:31

danielfullmer force-pushed the systemd-boot-assessment branch from 2f2e083 to c03ed62 Compare June 16, 2020 05:45

cole-h removed this from WIP in Staging Mar 22, 2021

cole-h reviewed Mar 23, 2021

View reviewed changes

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 21, 2021

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 17, 2021

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 16, 2022

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 16, 2022

RaitoBezarius mentioned this pull request Jun 22, 2023

[RFC 0153] Non-legacy boot NixOS NixOS/rfcs#154

Draft

misuzu mentioned this pull request Aug 23, 2023

DRAFT: nixos/systemd-boot: add boot counting feature #250937

Closed

12 tasks

dscottboggs mentioned this pull request Oct 29, 2023

Use fallback mechanism in GRUB to automatically recover unbootable system #26332

Open

RaitoBezarius mentioned this pull request Nov 5, 2023

Tracking issue: Boot security in NixOS #265640

Open

infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Dec 3, 2023

JulienMalka mentioned this pull request Dec 9, 2023

nixos/systemd-boot: init boot counting #273062

Merged

16 tasks

RaitoBezarius closed this Dec 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] nixos/systemd-boot: boot counting and automatic fallback #84204

[WIP] nixos/systemd-boot: boot counting and automatic fallback #84204

danielfullmer commented Apr 3, 2020 •

edited

grahamc commented Apr 11, 2020

flokli commented Apr 23, 2020

danielfullmer commented Apr 23, 2020 •

edited

flokli commented Jun 14, 2020

danielfullmer commented Jun 14, 2020

danielfullmer commented Jun 16, 2020

cole-h commented Mar 22, 2021 •

edited

cole-h left a comment

cole-h Mar 23, 2021 •

edited

stale bot commented Sep 21, 2021

RaitoBezarius commented Oct 17, 2021

stale bot commented Apr 16, 2022

RaitoBezarius commented Apr 16, 2022

danielfullmer commented Apr 16, 2022

lovesegfault commented Jul 26, 2022

dasJ commented Jul 26, 2022

domenkozar commented Aug 8, 2022

danielfullmer commented Aug 9, 2022

samueldr commented Aug 9, 2022

danielfullmer commented Aug 10, 2022

RaitoBezarius commented Jun 3, 2023

nixos-discourse commented Aug 8, 2023

nixos-discourse commented Dec 3, 2023

n8henrie commented Dec 8, 2023

RaitoBezarius commented Dec 9, 2023

	kernel_params = "systemConfig=%s init=%s/init " % (generation_dir, generation_dir)
	kernel_params = "init=%s/init " % generation_dir

[WIP] nixos/systemd-boot: boot counting and automatic fallback #84204

[WIP] nixos/systemd-boot: boot counting and automatic fallback #84204

Conversation

danielfullmer commented Apr 3, 2020 • edited

Motivation for this change

Things done

grahamc commented Apr 11, 2020

flokli commented Apr 23, 2020

danielfullmer commented Apr 23, 2020 • edited

flokli commented Jun 14, 2020

danielfullmer commented Jun 14, 2020

danielfullmer commented Jun 16, 2020

cole-h commented Mar 22, 2021 • edited

cole-h left a comment

Choose a reason for hiding this comment

cole-h Mar 23, 2021 • edited

Choose a reason for hiding this comment

stale bot commented Sep 21, 2021

RaitoBezarius commented Oct 17, 2021

stale bot commented Apr 16, 2022

RaitoBezarius commented Apr 16, 2022

danielfullmer commented Apr 16, 2022

lovesegfault commented Jul 26, 2022

dasJ commented Jul 26, 2022

domenkozar commented Aug 8, 2022

danielfullmer commented Aug 9, 2022

samueldr commented Aug 9, 2022

danielfullmer commented Aug 10, 2022

RaitoBezarius commented Jun 3, 2023

nixos-discourse commented Aug 8, 2023

nixos-discourse commented Dec 3, 2023

n8henrie commented Dec 8, 2023

RaitoBezarius commented Dec 9, 2023

danielfullmer commented Apr 3, 2020 •

edited

danielfullmer commented Apr 23, 2020 •

edited

cole-h commented Mar 22, 2021 •

edited

cole-h Mar 23, 2021 •

edited