sd-image: Fix Raspberry Pi 4 image builder #78090

cmacrae · 2020-01-20T09:55:55Z

Motivation for this change

These changes address the issues outlined in #73487.
They allow for rebuilds of system configuration to persist on the
Raspberry Pi 4. Without these changes, SD images generated from this
expression won't write their boot configs correctly, and thus the Pi
will just reboot to the installer configuration.

Along with this, also create /var/empty to allow for sshd to
start up correctly.

Things done

diamondburned · 2020-01-20T11:21:39Z

Is this tested? I'm not too confident in what I wrote...

colemickens · 2020-01-21T04:26:39Z

nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix

@@ -20,12 +20,23 @@
    firmwareSize = 128;
    # This is a hack to avoid replicating config.txt from boot.loader.raspberryPi
    populateFirmwareCommands =
-      "${config.system.build.installBootLoader} ${config.system.build.toplevel} -d ./firmware";
+      "${config.system.build.installBootLoader} ${config.system.build.toplevel} -d ./";


Why is this line even necessary?

As far as I can tell, it only runs for the initial SD card image build... and yet, my bootloader is properly configured everytime that I update the system, so I don't understand what this is doing one-time that is needed?

Yep, very good point. I think with the removal of ./firmware, it's probably not necessary at all.
I'll remove it and take it for a spin later when I get home

colemickens · 2020-01-21T04:27:09Z

This looks like what I've been using, but I haven't tested from scratch with a new SD Card. Also, I have one question about one of the lines.

cmacrae · 2020-01-21T08:29:23Z

Is this tested? I'm not too confident in what I wrote...

@diamondburned Yep, I've tested it on a fresh card. Boots fine and config changes work nicely 🙂

This looks like what I've been using, but I haven't tested from scratch with a new SD Card. Also, I have one question about one of the lines.
@colemickens Thanks for taking a look over this - I'll address your comments in the thread 👍

These changes address the issues outlined in NixOS#73487. They allow for rebuilds of system configuration to persist on the Raspberry Pi 4. Without these changes, SD images generated from this expression won't write their boot configs correctly, and thus the Pi will just reboot to the installer configuration. Along with this, also create '/files/var/empty' to allow for sshd to start up correctly.

colemickens · 2020-01-22T07:58:53Z

There's another issue that I've had to work around and I've seen others mention - this module is still not really able to be included by the end user in a day-to-day usage device - you can't remove the installation-device profile and thus can't remove the nixos user, enable SSH at boot, etc.

cc: @samueldr

samueldr · 2020-01-22T19:11:07Z

nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix

-    # As the boot process is done entirely in the firmware partition.
-    populateRootCommands = "";
+    # /var/empty is needed for some services, such as sshd
+    populateRootCommands = "mkdir -p ./files/var/empty";


If this is required, this shouldn't be handled here. This should be fixed upstream in NixOS so that when it is missing, it works. I think this explains an issue I saw on another board, but did not investigate.

At the very least, if it's not part of some tmpfiles declaration, it should be part of the first boot commands in the generic sd image builder.

I thought this would be handled in the activation script:

https://github.com/NixOS/nixpkgs/blob/staging/nixos/modules/system/activation/activation-script.nix#L184-L198

iirc Pi images are treated as if they're live CDs, which doesn't run this. I'm not sure though, I just know that without those commands, there wouldn't be a /var/empty.

Yeah that sounds right, might need this for the other rpi images too.

yes, I've used this same for rpi3:

systemd.tmpfiles.rules = [ "d /var/empty - - - -" ];

I wonder what's the failure mode here. Why isn't it required for other systems than the ARM images?

On rpi3 I'm also experiencing sshd crashing with Missing privilege separation directory: /var/empty. Only on the first boot. If i reboot once, it's fine.
On the first boot when it's not working dmesg | grep /var/empty gives me:

[ 8.485396] stage-2-init: mkdir: cannot create directory '/var/empty': No space left on device [ 8.536393] stage-2-init: find: '/var/empty': No such file or directory [ 8.545242] stage-2-init: chmod: cannot access '/var/empty': No such file or directory [ 8.556068] stage-2-init: chown: cannot access '/var/empty': No such file or directory

Seems like there is no space left on the sd card at this point? Maybe because resizing hasn't have happened yet?
On the second boot there is nothing in dmesg related to /var/empty.
Should i open a bug report for this?

Oh, that could be the root cause of this issue I guess. I would rather this gets fixed appropriately rather than patched over.

If you open a bug, it is not likely sshd-specific, but really this one issue. And please do, cc me once you do.

I vote for merging this immediately, since a hack that fixes broken sshd is better than broken sshd.

t. person that has wasted the last hours trying to log into sshd on his raspberrypi.

samueldr · 2020-01-22T19:24:25Z

@colemickens:

This is the installer image, as described by the path of the file. It shouldn't be included on a running end-user system.

We don't usually provide hardware-specific configurations in the Nixpkgs repo. For this there is nixos-hardware where we probably should begin adding SBCs too.

But you're right that this is some kind of expectations mismatch. The end-user wants to run the raspberry pi system on their sd card, so why can't they include sd-image-raspberrypi4? Well, that's the installer/ profile.

In fact, that profile file, I announce, will be removed from 20.03 once 20.03 is branched off.

Noting again: as previously stated, it will be kept in unstable up until the moment mainline Linux runs on the Raspberry Pi 4. Once mainline boots on the Pi 4, the generic aarch64 image will be the image for both the pi 3 and 4.

kwohlfahrt · 2020-03-03T22:32:09Z

nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix

+		"/boot" = {
+			device = "/dev/disk/by-label/FIRMWARE";
+			fsType = "vfat";
+		};


This was changed in #62462, I think mostly because /boot can get quite big if it stores multiple versions of the kernel. @samueldr, you authored the original PR and you've commented here, but have not addressed this particular difference. Any thoughts?

VanCoding · 2020-04-21T22:46:37Z

@samueldr On my Raspberry PI, when I change boot.loader.raspberryPi.firmwareConfig and run nixos-rebuild switch the config.txt on the /dev/sda1 partition does not get changed at all. Is this what this pull request tries to solve?

Shouldn't the content of /dev/sda1 be replaced on every nixos-rebuild switch?

I stumbled upon this while trying to bring a RPI Kisok to work using NixOS and also requested for help here: https://discourse.nixos.org/t/raspberry-pi-3-official-touch-screen-not-working/5735/5?u=vancoding

kwohlfahrt · 2020-04-22T11:13:26Z

I think this just the installer image, so it won't affect running systems (i.e. nixos-rebuild switch). I also think that your suggestion about replacing the firmware partition on configuration change (though mine is on /dev/mmcblk0p1) is reasonable but not related to this PR. I'll hopefully get around to implementing that soon...

VanCoding · 2020-04-22T13:43:29Z

@kwohlfahrt Ah, okay.

Yeah, having the same configuration for image generation and updating is what I'm trying to achieve.

I want to manage many Raspberry PIs, which should have all the same configuration (which gets updated from time to time). And adding an additional Raspberry PI should be as easy as taking this configuration and generating an image for it.

But that's OT, sorry.

samueldr · 2020-06-21T02:53:00Z

If you can reproduce your issues, can you validate that, with #90119 in the commit you are using to build the image, the installer image works as expected?

It should now:

Mount the boot partition correctly
Use a less confusing label for the boot partition

Profpatsch · 2020-06-22T17:40:46Z

The build of the image stopped working, but I didn’t get my Raspberri Pi 4 Model B (armv7l) to accept ssh (I don’t have a mini-HDMI cable here so I can’t attach a monitor). I decided to just use raspian and try to nixos-infect the system instead, because I have no idea how to debug this.

…

On Sun, Jun 21, 2020 at 4:53 AM Samuel Dionne-Riel ***@***.***> wrote: If you can reproduce your issues, can you validate that, with #90119 <#90119> in the commit you are using to build the image, the installer image works as expected? It should now: - Mount the boot partition correctly - Use a less confusing label for the boot partition — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#78090 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYB5ZSZ5SRUIZZCRKXJEOTRXVYZPANCNFSM4KJBLNWQ> .

Profpatsch · 2020-06-22T17:41:53Z

s/stopped working/stopped failing/ I don’t know if the system boots at all or if the sshd crashes or if my config is just plain wrong. On Mon, Jun 22, 2020 at 7:40 PM Philip Patsch <philip.patsch@tweag.io> wrote:

…

The build of the image stopped working, but I didn’t get my Raspberri Pi 4 Model B (armv7l) to accept ssh (I don’t have a mini-HDMI cable here so I can’t attach a monitor). I decided to just use raspian and try to nixos-infect the system instead, because I have no idea how to debug this. On Sun, Jun 21, 2020 at 4:53 AM Samuel Dionne-Riel < ***@***.***> wrote: > If you can reproduce your issues, can you validate that, with #90119 > <#90119> in the commit you are > using to build the image, the installer image works as expected? > > It should now: > > - Mount the boot partition correctly > - Use a less confusing label for the boot partition > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#78090 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAYB5ZSZ5SRUIZZCRKXJEOTRXVYZPANCNFSM4KJBLNWQ> > . >

samueldr · 2020-06-22T21:36:29Z

Note that as with all images using the installer profile, the SSHD will not start by default. Furthermore, it would require additional credentials on the device. This is all assuming you are not using a custom build.

Now, what I understand from this discussion is that this is a first-boot only issue, and is not specific to the raspberry pi. Re-doing and fixing the hack here is not a solution, it needs to be done for all sd images.

BUT, note that we just added a patch that changes something with how the images are built, and it's possible it also fixes that issue with starting sshd since it may now have enough space on the partition during boot for this to work.

The proper solution is the one from @danbst

                systemd.tmpfiles.rules = [
                    "d /var/empty - - - -"
                ];

Which I think would be appropriate to add to the generic sd-image expressions, at worst it's a no-op, at best here it'll fix that specific issue.

I do not have a setup with a reproduction of the issue, so I'm not in the best situation to validate that issue.

samueldr · 2020-06-22T21:37:47Z

The build of the image stopped working, but I didn’t get my Raspberri Pi 4
Model B (armv7l) to accept ssh (I don’t have a mini-HDMI cable here so I
can’t attach a monitor).

The Raspberry Pi 4 Model B is an aarch64 device, but since you said "(armv7l)" I am assuming this comes from raspios (née Raspbian) which is running on 32 bit. I don't know that you can successfully infect a 32 bit ARM system with an aarch64 NixOS system.

Profpatsch · 2020-06-23T12:58:00Z

The Raspberry Pi 4 Model B is an aarch64 device, but since you said "(armv7l)" I am assuming this comes from raspios (née Raspbian) which is running on 32 bit.

Ah, I was super confused by that. So uname shows armv7l if it’s running on a 32bit system, which raspios is, but the system itself is 64 bit.

I think I’m gonna try again with the /var/empty fix, I only used the PR that was merged into master last week.

I don't know that you can successfully infect a 32 bit ARM system with an aarch64 NixOS system.

I wonder, nixos-infect should not require running any 64bit executables before reboot, and after reboot it has a 64bit kernel … though the question is whether the boot partition stuff is set up correctly by nixos-infect anyway, and if it’s not I would be in the same situation as with the SD image.

Thank you for the help!

urbas

The RPi4 SD image has since moved on quite a bit and the sshd bug also seems to have been fixed. We might want to consider closing this PR at this point.

matthewbauer · 2021-02-01T21:37:39Z

Feel free to reopen if it's still relevant.

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jan 20, 2020

colemickens reviewed Jan 21, 2020

View reviewed changes

cmacrae force-pushed the raspbery_pi_4/fix_sdimage_builder branch from 3470a8a to febcf75 Compare January 21, 2020 08:33

samueldr requested changes Jan 22, 2020

View reviewed changes

kwohlfahrt reviewed Mar 3, 2020

View reviewed changes

Robertof mentioned this pull request May 14, 2020

Error while copying store paths to image Robertof/nixos-docker-sd-image-builder#1

Closed

angerman mentioned this pull request Jun 11, 2020

Fix some pitfalls from the Raspberry Pi 4 specific sd image #90119

Merged

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

urbas self-requested a review January 30, 2021 16:41

urbas reviewed Feb 1, 2021

View reviewed changes

matthewbauer closed this Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sd-image: Fix Raspberry Pi 4 image builder #78090

sd-image: Fix Raspberry Pi 4 image builder #78090

cmacrae commented Jan 20, 2020

diamondburned commented Jan 20, 2020

colemickens Jan 21, 2020

cmacrae Jan 21, 2020

colemickens commented Jan 21, 2020

cmacrae commented Jan 21, 2020

colemickens commented Jan 22, 2020

samueldr Jan 22, 2020

matthewbauer Jan 28, 2020

diamondburned Jan 29, 2020

matthewbauer Jan 29, 2020

danbst Feb 3, 2020

samueldr Feb 3, 2020

DavHau Feb 3, 2020 •

edited

samueldr Feb 4, 2020

Profpatsch Jun 20, 2020

Profpatsch Jun 20, 2020

samueldr commented Jan 22, 2020

kwohlfahrt Mar 3, 2020

VanCoding commented Apr 21, 2020

kwohlfahrt commented Apr 22, 2020

VanCoding commented Apr 22, 2020

samueldr commented Jun 21, 2020

Profpatsch commented Jun 22, 2020 via email

Profpatsch commented Jun 22, 2020 via email

samueldr commented Jun 22, 2020

samueldr commented Jun 22, 2020

Profpatsch commented Jun 23, 2020

urbas left a comment

matthewbauer commented Feb 1, 2021

sd-image: Fix Raspberry Pi 4 image builder #78090

sd-image: Fix Raspberry Pi 4 image builder #78090

Conversation

cmacrae commented Jan 20, 2020

Motivation for this change

Things done

diamondburned commented Jan 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colemickens commented Jan 21, 2020

cmacrae commented Jan 21, 2020

colemickens commented Jan 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DavHau Feb 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samueldr commented Jan 22, 2020

Choose a reason for hiding this comment

VanCoding commented Apr 21, 2020

kwohlfahrt commented Apr 22, 2020

VanCoding commented Apr 22, 2020

samueldr commented Jun 21, 2020

Profpatsch commented Jun 22, 2020 via email

Profpatsch commented Jun 22, 2020 via email

samueldr commented Jun 22, 2020

samueldr commented Jun 22, 2020

Profpatsch commented Jun 23, 2020

urbas left a comment

Choose a reason for hiding this comment

matthewbauer commented Feb 1, 2021

DavHau Feb 3, 2020 •

edited