@Mic92 I didn't want to introduce a build time dependency on python3 since the current nixos kernel builds fine without my patch. Was trying to find what config setting calls bpf_helper_docs.py but it turns out my current kernel config works fine without that workaround too.

I've ended up simplifying the ld-config line and dropping bpf_helper_docs.py.

I guess gawk and zstd in nativeBuilInputs is okay. But where is the python3 patching now ?
Is python shebang patched by patchShebangs in the last update ?

I dropped that change, didn't want to introduce a dependency on python3 just to build the kernel, especially since it's not required for the default nixos builds.

#105777 (comment)

Ok, everything LGTM :-)

sry if I missed it: why do we need gawk ?

It's used in ld-config. The change just makes it an explicit dependency and converts the sed to patchShebang -- it was already used before.

(See #105777 (comment))

looks good. I wanna merge but I wonder if it shouldn't go into staging instead ? that's what I was asked in a similar PR #106395

Users don't directly benefit from this change and it causes a moderate number of rebuilds, I think staging is appropriate.

Since it's going into staging, I'll go ahead and configure zstd compression as well.

What's a good way to change the boot.initrd.compressor depending on the kernel version?

Would this work?

default = (if `lib.versionOlder kernel.version "5.9" then "gzip -9n" else "${lib.getBin pkgs.zstd}/bin/zstd")

I'm not too familiar with nixpkgs so I don't understand if that's evaluated late enough.

ztd -> zstd
`lib -> lib

I'd flip the conditional, "you need at least 5.9 for zstd" is clearer in my head.

The zstd compressor should probably be given a higher level. ~10 is appropriate I think; that should still compress in under a second and doesn't impact decompression speed significantly.

Keep in mind that what you're now touching is NixOS. It uses the module system for combination/evaluation and you need to use boot.kernelPackages.kernel to get the boot kernel declared in the user's config module (or the default).

The levels are not directly comparable though. zstd defaults to 3 which compresses both faster and with a better ratio than gzip -9n.

Based on fedora benchmarks it seems that 10 doesn't yield significantly more savings compared to the extra time spent compressing. (Arch/Fedora packaging uses 19 however, but that's because it's worth the extra build server time).

A few more figures here: https://community.centminmod.com/threads/round-4-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.18669/

Keep in mind that they're compressing a 4.7GiB cpio in those tests. My /run/current-system/initrd is ~16M (compressed).

Level 10 does yield a ~10% reduction over 3 according to your links and the squashfs benchmark shows similar results. It does fall off dramatically from 10-15 though and anything higher is way to slow and outclassed by lzma and others.

The Fedora benchmark's ~10MiB/s is rather slow compared to the squashfs benchmark's 10y/o mid-range laptop's ~28MiB/s but even that'd only take a few seconds for a few dozen MiB large initrd which is still acceptable and worth it for 10% space savings IMO.
These space savings actually matter here; I have run out of space in boot partitions multiple times with NixOS due to old generations and I can't resize it with my setup. I imagine many of us might be in similar situations.

Ready for another review, sorry for the delay. Should this be put in the next release notes too?

The regular LTS kernel is not affected as expected (config built with one has a gzip'd initrd) and my 5.10 system boots without problems using the zstd initrd.

-T defaults to 1. We could set it to -T$NIX_BUILD_CORES but I'm not sure if it's worth it, given the small filesize / compression time.

Why zstd instead of lz4?
https://facebook.github.io/zstd/:

Decompress:
zstd 1.4.5 --fast=5 2420 MB/s
lz4 1.9.2 4530 MB/s

https://smackerelofopinion.blogspot.com/2019/09/boot-speed-improvements-for-ubuntu-1910.html:

The Ubuntu kernel team ran several experiments benchmarking several x86 configurations using the x86 TSC (Time Stamp Counter) to measure kernel load and decompression time
As media gets faster, the load time difference between GZIP, LZ4 and LZO diminishes and the decompression time becomes the dominant speed factor with LZ4 the clear winner.
For Ubuntu 19.10 Eoan Ermine, LZ4 will be the default decompression for x86, ppc64el and s390 kernels and for the initramfs too.

(Note: LZ4 is by the same author as ZSTD.)

ZSTD strikes a nice balance between compression ratio and speed. LZ4 is extremely fast but won't compress as well.

Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
When we switched from an lzma compressed initramfs to a zstd compressed initramfs
decompression time shrunk from 27 seconds to 8 seconds.

The zstd compressed kernel is smaller than the gzip compressed kernel but larger
than the xz or lzma compressed kernels, and it decompresses faster than
everything except lz4.

https://lwn.net/Articles/816054/

I worry that zstd will be a bottleneck on fast drives.
https://facebook.github.io/zstd/:

If you have an initrd/kernel that's big enough to make a difference there, you've got another problem :) zstd is already about five times faster than gzip which was the default so far.

As indicated by Atemu #105777 (comment), the initrd filesize matters more, since we don't want to fill up the boot partition. You're always free to change the boot.initrd.compression setting for your own setup, but I think zstd is a more balanced option for the distribution default.

My initrd is ~35MiB. According to FB's benchmark, that would take ~22ms to decompress with zstd and ~8ms with lz4.

If you have a config where 14ms would be a significant part of the boot process, I'd love to see it.

You're always free to change the boot.initrd.compression setting for your own setup

Actually, I don't think you are; that's an internal option. I don't see much reason for it should stay one though.

-T defaults to 1.

Last time I checked, it was supposed to default to -T$(nproc).

We could set it to -T$NIX_BUILD_CORES but I'm not sure if it's worth it, given the small filesize / compression time.

I did test to manually set it to -T12 and an insanely high level to have an observable compression time but it still only used one core. I don't think it'd be worth bothering either though.

Hmm I guess it was undocumented? I was using this locally before:

https://github.com/archseer/snowflake/blob/master/hosts/hyrule/default.nix#L40

-T#, --threads=#: Compress using # working threads (default: 1). If # is 0, attempt to detect and use the number of physical CPU cores.

the initrd filesize matters more, since we don't want to fill up the boot partition.

Okay, that is fine.

If you have a config where 14ms would be a significant part of the boot process, I'd love to see it.

Just so you understand the rational for my personal choice of using lz4: EFI system partitions are typically 260 MiB, so I would rather have the performance improvement over size on my system.

You're always free to change the boot.initrd.compression setting for your own setup

Actually, I don't think you are; that's an internal option. I don't see much reason for it should stay one though.

#97145

#97145

Nice find! Made a comment there to add -10 to zstd default args.

https://github.com/facebook/zstd/releases/tag/v1.4.7

@TredwellGit Out of scope for this PR. We don't care what version the CLI tool is, it just has to produce a file in ZSTD format that the kernel can read.

#107132

I posted it just in case any of the new changes would influence the compression options chosen.

for some reason, this was not closed automatically after #107703 was merged.. closing manually