I always get an failed to move initrd secrets into place error. This even happens if I have no initrd secrets on that host.

@Kloenk That should be fixed now.

I also fixed another issue that would happen when copyKernels == false, but I still haven't tested that configuration. My only system that uses GRUB has an encrypted rootfs, and I haven't figured out how to test GRUB in a NixOS VM, because they normally don't use a bootloader.

I also removed internal = true from boot.secrets.initrd, which was the reason it didn't appear in the manual and @mdorman thought it was undocumented.

It still fails. I don't know what the problem is, and I don't have an clue on where to start debugging.

The /boot/kernels directory contians an <hash>-initrd-secrets file, but this initrd is use by 3 generations bevore.

The log output I get from the nixos-rebuild switch ... command is:

updating GRUB 2 menu...
4 blocks
4 blocks
4 blocks
4 blocks
4 blocks
4 blocks
failed to move initrd secrets into place

The non initrd-secrets machine is still building, will post if it also fails tomorrow.

Are you sure you aren't using a module that uses the initrd secrets functionality under the hood (e.g. the initrd SSH module)? nixos-option boot.initrd.secrets might reveal something.

Oh, I see now. The current code assumes that all generations either do or don't have initrd secrets enabled, which obviously isn't the case when you first enable the option.

One disadvantage of removing support for extraInitrd immediately is that I guess it will immediately break boots of all old generations that use it as soon as you perform a rebuild, so you can't roll back if you mess up migration to initrd secrets.

Are you sure you aren't using a module that uses the initrd secrets functionality under the hood (e.g. the initrd SSH module)? nixos-option boot.initrd.secrets might reveal something.

The initrd ssh module is the only secret I use. On my local PC (at the moment building krita) I don't have initrd ssh. On my server I have ssh, and there it is failing

One disadvantage of removing support for extraInitrd immediately is that I guess it will immediately break boots of all old generations that use it as soon as you perform a rebuild, so you can't roll back if you mess up migration to initrd secrets.

The existing implementation already has that problem. The current generation's extraInitrd is applied to all generations. If you disable extraInitrd in the current generation, it gets disabled in all generations.

I rewrote the implementation to be more similar to how systemd-boot does it. It should now work correctly when only some of the generations have secrets.

I have fixed all the known issues, but I think it still needs more testing, particularly on systems with the a single boot/root partition.

I finaly got around to deploy it on my initrd-ssh-server machines. It works there, but only after I did a nix-collect-garbage -d, because of old generations of my systems, for whom it tried to build initrd-secrets for

Seems like an instance of #85563. Was the error just that the extraInitrd file no longer existed, or was the initrd secrets code somehow running on the old generations and failing? If the former, we should maybe just ditch the advice to rm it to save people problems rolling back.

As far as i understood it, it tried to create an initrd for a previos generation. The config of this generation looked for the initrd secret on another path, and so it failed

Like @emilazy said, that sounds like an issue with the current initrd secrets implementation, not specific to GRUB.

If the former, we should maybe just ditch the advice to rm it to save people problems rolling back.

The documentation doesn't currently suggest that users delete the initrd file. I think rollbacks (ie. nixos-rebuild switch --rollback) will still work as long as the initrd file still exists, depending on whether nixos-rebuild runs the bootloader install script in that case. If you directly boot into an older generation, the extra initrd will be missing. There is really no way to fix this, because there is no way to know what initrd file was configured for an older generation.

So it seems like everything is working as intendet. Maybe write something about old profiles in the release note, so nobody runs into the same problem as me?

I added some clarification to the changelog, describing the issue I mentioned in my previous comment. It doesn't really talk about the issue @Kloenk ran into, because that isn't related to GRUB or this PR specifically.

Tested on my server as well.

it doesn't seem to rebuild the secrets file if the contents of the secrets change
in my case, i had accidentally set a secret to the wrong file, but upon fixing it and rebuilding, the secrets file remained unchanged

I'm not sure how that could happen. The initrds for all generations should be generated each time you activate a new configuration. Could you provide more details? Did you change the path to the secret file between the generations, or just its contents?

The second commit also causes the initrd-network-ssh test to fail. It looks like the secret is not copied into the initramfs.
This can be seen e.g. in the output of this hydra build:

server # chmod: /etc/ssh/sh_host_ed25519_key: No such file or directory
server # Unable to load host key: /etc/ssh/sh_host_ed25519_key
server # sshd: no hostkeys available -- exiting.

I'm not totally sure yet, but I think this is a bug in the test. Before this PR, GRUB did not support initrd secrets, so the secrets were copied to the Nix store. Now, they need to be copied by the bootloader install script, which does not run because the test does not use a bootloader.

Setting virtualisation.useBootLoader might be the right solution, but it causes the build to fail for me.

See #91744

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/full-encrypted-nixos-system-on-legacy-boot-with-secrets-and-remote-unlock-for-unstable-20-03/8279/1