sd-image-*: Use F2FS instead of ext4 for root FS #106031

jevankovich · 2020-12-06T01:04:39Z

Motivation for this change

F2FS has better behavior on SD cards and can improve performance and endurance over ext4. Closes #42242

Things done

Added a partition expansion option to filesystems that expands the extent of the partition during stage-1, alongside filesystem resizing. This is needed because an F2FS partition cannot be resized after mounting it r/w.
Added a make-f2fs-fs.nix module.
Changed all sd-image-* modules to use F2FS.
- This involved reverting the /boot/firmware//boot split. /boot is now on FAT since u-boot doesn't support F2FS
- The size of the /boot partition was increased to 256 MB (248, really, so the end of the partition is at the 256 MB boundary of the image)

Things to note:

This change uses sload.f2fs to populate the root FS which seems to be a lot slower than the equivalent for ext4. Expect image builds to take about 5 minutes longer than before.
- It's possible there's a better option here. Unsure
Likewise, resizing an F2FS parition is fairly slow, so the first boot where the root partition gets resized can take a while. I'm not sure if this is dependent on SD card size or if it's related to the number of blocks that are allocated.
- If it's related to # of blocks allocated, maybe it would make sense to enable file compression at the FS level for these images.
F2FS partitions cannot be shrunk, so users who have more complex partitioning needs may not want this change.
- It may be possible to add an option to the partition expanding feature that expands to all but the top N Bytes of the disk. It's always possible to expand more later if desired.

The images can be downloaded here, built on top of nixos-unstable:

Generic aarch64 image
Raspberry pi 4 image
Generic armv7l image (Currently building)
Raspberry pi 0/1 image (Currently building)

TEMPORARY, REVERT BEFORE PR

This reverts commit bb0407ec466ee130bb9f6bdb497b908c5af38d00.

jevankovich · 2020-12-06T01:08:02Z

Unsure who would be good for review.

@kisik21 created the original issue.

@samueldr seems to contribute to raspberry pi related things and might know people who should review.

jevankovich · 2020-12-06T02:23:45Z

Something I missed before while testing the images. Partition resizing seems to be failing now. Not sure what would've caused it.

The error I'm getting is Error: Device size is not sufficient for F2FS volume, more segment needed =4977.

It seems to be coming from here, but I don't know enough about F2FS to say with confidence what's going on or how to fix it.

vikanezrimaya · 2020-12-07T10:18:34Z

I'll try this one more time on my sacrificial SD card, now building from source. I assume you don't have a binary cache, but I happen to have one on Cachix and could push the images there to ease testing for other people.

vikanezrimaya

The manual fails to build. Derivation /nix/store/bzs66hzyiqy64bdfal1nhgsawwyv0qga-nixos-manual-combined.drv fails. Here's the code to reproduce (don't forget to enable flake-support - --option experimental-features "flake nix-command")

((builtins.getFlake "github:jevankovich/nixpkgs/f2fs-image").lib.nixosSystem { modules = [
  ({ config, lib, pkgs, modulesPath, ... }: {
    imports = [
      (modulesPath + "/installer/cd-dvd/sd-image-raspberrypi4.nix")
    ];
    # Explicitly set the architecture
    nixpkgs.localSystem.system = "aarch64-linux";
    # Let's have a bigger boot size! I want to keep lots of boot configurations
    # and usually I have a /boot sized around 500-1000MB to be safe
    sdImage.bootSize = 768;
    # Enable UART for Raspberry Pi 4 so I won't have to connect it to a TV
    boot.kernelParams = [ "console=ttyAMA0,115200" "console=tty1" ];
  })
]; }).config.system.build.sdImage

urbas · 2021-01-08T06:45:41Z

RPi 4 (with newer firmware) allows to boot from USB storage. This means one can boot from HDDs and SSDs, for which I'd prefer to stick with ext4. Could you add an option to keep using ext4?

urbas · 2021-01-08T06:53:57Z

nixos/modules/tasks/filesystems.nix

@@ -107,6 +107,17 @@ let
        '';
      };

+      autoExpand = mkOption {


This looks like a new feature. It would be easier to review this PR and move it forward if you isolated this feature and opened a separate PR for it.

samueldr

All in all, I am against changing the default here. Those two general broad changes

the opaque "raspberry pi firmware" partition concept (as much as I hate it)
ext4 as the default

What I would be for is figuring out a strategy to allow the user to choose a filesystem they desire. Or totally review the user story on ARM from scratch, rather than going one step back, and one step to the side.

Though, with that said, I think I like the approach for the partition expansion, which is totally independent from the filesytem expansion. Might need a good review, which I haven't done. I'd like to see a PR for that change in isolation if not asking too much.

samueldr · 2021-01-08T08:36:53Z

nixos/modules/installer/cd-dvd/sd-image.nix

-      "/boot/firmware" = {
-        device = "/dev/disk/by-label/${config.sdImage.firmwarePartitionName}";
+      "/boot" = {
+        device = "/dev/disk/by-label/${config.sdImage.bootPartitionName}";


There is a reason it is /boot/firmware, it is because it is an "opaque partition" that is used exclusively to hold u-boot for the raspberry pi. This partition has to be removed for Rockchip boards as their bootloader installed to an arbitrary location encroaches on that partition. This would break usage on Rockchip boards.

Additionally, having a separate boot partition is painful, unless we enlarge it enough to hold enough generations. (Which I see you have done).

Though, really it would be preferable to not have /boot as another partition by default.

nixos/modules/system/boot/stage-1-init.sh

samueldr · 2021-01-08T08:51:39Z

(Additional context)

I say "as much as I hate it" for the opaque firmware partition, because I would much prefer to not have it at all. The only reason it is there is because it is really hard to "simply dd" the firmware (u-boot) on the storage for raspberry pi, compared to other boards. This is because it requires a tangible partition, rather than arbitrary code at an arbitrary offset.

This "firmware" partition is a wart, I don't like it, and I'd much prefer to see only one partition, the rootfs.

Accidentally described partition expansion as filesystem expansion Co-authored-by: Samuel Dionne-Riel <samuel@dionne-riel.com>

jevankovich · 2021-01-08T15:47:26Z

On /boot vs /boot/firmware

I was unaware of the situation with Rockchip boards. The reason I had to take this step backwards was because U-boot doesn't have drivers for f2fs, so the kernel needs to be on a partition that u-boot can read. The path of least resistance here was to move the kernel onto the fat32 partition.

It's possible f2fs is incompatible with a generic ARM image until (or unless) f2fs support is added to u-boot.

Maybe it makes sense to have a Pi specific f2fs image alongside the generic arm/ext4 image? Extra maintenance burden to keep them both working, though.

Crazy alternative - Have a very minimal image that allows the user to partition the rest of the media as desired so they can choose the appropriate fs for their system. Seems better than having an image for every fs someone could want but I wouldn't have time to take that on.

On partition expansion as a separate PR

Agreed, I'll pull that together.

For context as to why I needed to add it in the first place: After root is mounted rw, the f2fs tools were not allowing resizing of the filesystem, so the partition and filesystem had to be grown in stage-1. At least, I think that's what's going on. I'm not exactly clear on the different boot stages.

Could be useful to simplify some of the startup scripts in SD images and may be good for VPS images.

petersjt014 · 2021-04-29T07:31:49Z

Just tested this on baremetal (rpi 3B). I had to run dd with sudo for some reason (not sure if that's related), but it boots fine. Resizes in only about a minute too.

I've noticed two problems: one, that the display goes black (as if the pi had been disconnected from the display) on an erratic interval. This could be a cable problem or something on my end or the fact that I'm undervolting, but I haven't seen it before. The second problem happens when I try to run nixos-generate-config:

writing /etc/nixos/hardware-configuration.nix...
write_file '/etc/nixos/hardware-configuration.nix' - sysopen: Permission denied at /run/current-system/sw/bin/nixos-generate-config line 597.

I got a similar error when trying to start wpa_supplicant, and while doing a few other things.

Also, nixos-rebuild fails due to a lack of space, apparently, and some other actions involving the store do the same. Installing with nix-env -iA works fine though.

edit: and now my display fails to detect it at all, so that sucks. I'll airblast the dust out of it later today, see if that helps--if not I can possibly start up ssh on it while flying blind. If I can get back into it I'll be happy to test anything anyone wants with it.

urbas · 2021-05-01T08:40:22Z

nixos/modules/installer/cd-dvd/sd-image.nix

@@ -16,7 +12,7 @@
 with lib;

 let
-  rootfsImage = pkgs.callPackage ../../../lib/make-ext4-fs.nix ({
+  rootfsImage = pkgs.callPackage ../../../lib/make-f2fs-fs.nix ({


This PR is still changing the default FS without giving an option to choose the FS.

This is a blocker from my point of view. I would vote to keep keep the default ext4 and add an option to choose f2fs.

Moreover, the PR contains mulriple other changes. Here are the other changes in this PR I was able to identify and believe can be split out:

renaming firmwareXYZ options to bootXYZ

changing the populateRootCommands option

changing the /boot/firmware filesystem to /boot

moving the auto-expand option to the stage-1-init.sh script (and adding the autoExpand option to the filesystems module)

Could you please create a PR where the only change is making the root FS type configurable? The other changes should go into individual PRs.

jevankovich · 2021-05-01T13:50:50Z

I haven't looked at this in months. Quite surprised by the recent attention.

I'm not sure what the right process is here, so I'm going to close the PR. If anyone wants to pick it back up they can open new ones and use this as reference.

voidus · 2022-09-10T00:38:44Z

I've started picking this up in https://github.com/voidus/nixpkgs/tree/sd-image-f2fs
I got it built with f2fs as the file system but I still need to deal with the fact that U-boot doesn't speak f2fs, I don't think I've 100% understood what the issue here is.

I guess I'll create a new PR in the next few days to discuss these things.

jevankovich added 9 commits December 5, 2020 15:04

Move partition expanding into stage-1 with fs resize

47c0020

Change aarch64 image to use kernel 4.19

c4d8b78

TEMPORARY, REVERT BEFORE PR

Change aarch64 sd image to use f2fs

2651e8f

Revert "Change aarch64 image to use kernel 4.19"

6c2cdac

This reverts commit bb0407ec466ee130bb9f6bdb497b908c5af38d00.

Port pi 4 image to F2FS sd image

e320307

Increase size of /boot partition so kernel upgrades succeed

dc0e250

Removed dependency on faketime since f2fs tools have -T option

eb49460

Removed dependency on fakeroot for make-f2fs-fs.nix

87579a7

Ported over other sd-images to f2fs

36e949f

ofborg bot added 6.topic: nixos 8.has: module (update) labels Dec 6, 2020

Remove trailing whitespace from filesystems.nix

e95eefb

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Dec 6, 2020

vikanezrimaya suggested changes Dec 7, 2020

View reviewed changes

urbas reviewed Jan 8, 2021

View reviewed changes

samueldr requested changes Jan 8, 2021

View reviewed changes

Fix wording in nixos/modules/system/boot/stage-1-init.sh

7bb6c54

Accidentally described partition expansion as filesystem expansion Co-authored-by: Samuel Dionne-Riel <samuel@dionne-riel.com>

ofborg bot added the 2.status: merge conflict label Jan 8, 2021

petersjt014 requested review from urbas and samueldr April 29, 2021 00:35

urbas suggested changes May 1, 2021

View reviewed changes

jevankovich closed this May 1, 2021

samueldr mentioned this pull request Sep 2, 2023

Support for building sd-image with a FS other than Ext4 #252868

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sd-image-*: Use F2FS instead of ext4 for root FS #106031

sd-image-*: Use F2FS instead of ext4 for root FS #106031

jevankovich commented Dec 6, 2020 •

edited

jevankovich commented Dec 6, 2020

jevankovich commented Dec 6, 2020

vikanezrimaya commented Dec 7, 2020

vikanezrimaya left a comment

urbas commented Jan 8, 2021

urbas Jan 8, 2021

samueldr left a comment

samueldr Jan 8, 2021

samueldr commented Jan 8, 2021

jevankovich commented Jan 8, 2021

petersjt014 commented Apr 29, 2021 •

edited

urbas May 1, 2021

jevankovich commented May 1, 2021

voidus commented Sep 10, 2022

sd-image-*: Use F2FS instead of ext4 for root FS #106031

sd-image-*: Use F2FS instead of ext4 for root FS #106031

Conversation

jevankovich commented Dec 6, 2020 • edited

Motivation for this change

Things done

Things to note:

jevankovich commented Dec 6, 2020

jevankovich commented Dec 6, 2020

vikanezrimaya commented Dec 7, 2020

vikanezrimaya left a comment

Choose a reason for hiding this comment

urbas commented Jan 8, 2021

urbas Jan 8, 2021

Choose a reason for hiding this comment

samueldr left a comment

Choose a reason for hiding this comment

samueldr Jan 8, 2021

Choose a reason for hiding this comment

samueldr commented Jan 8, 2021

jevankovich commented Jan 8, 2021

On /boot vs /boot/firmware

On partition expansion as a separate PR

petersjt014 commented Apr 29, 2021 • edited

urbas May 1, 2021

Choose a reason for hiding this comment

jevankovich commented May 1, 2021

voidus commented Sep 10, 2022

jevankovich commented Dec 6, 2020 •

edited

petersjt014 commented Apr 29, 2021 •

edited