New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nixos: compress make-ext4-fs with zstd #75592
Conversation
Difference between bzip2 and zstd
|
I've done the work to use zstd in another branch and the difference is staggering. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As it is, this breaks the API of make-ext4-fs.nix
.
I do not know what is our guarantee with internal~ish builders like those.
This could be fixed by adding a compressed ? false
param to the builder. the main difference would be that bzip2
invocation is not used, but rather a cp
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Ugh, forgot to tick request changes)
I don't think that the following is an issue with which we should concern ourselves right now, but to be kept in mind for the eventual rewrite. Writing a temporary file is costly. It could result in failing builds if the For the second part (streaming into the final image), it would need metadata written in $out (so $out would not be directly an image file) that tells the image builder what is the size of the image. For the first part, (compression) I guess it depends on what compresses the image file. Are there ways that it can be done in-place? Though, if it's writing in a A final strategy for the future could be to merge multiple snippets into one big honking derivation that does all the work. The main drawback would be that the derivation is less flexible, and takes more time to build. A well architected solution could possibly do this with an option. |
d378910
to
21d7243
Compare
21d7243
to
5f5a7b1
Compare
e41ae61
to
3891832
Compare
73ecb62
to
8fdf1a1
Compare
8fdf1a1
to
70c5a78
Compare
Motivation for this change
The AArch64 build has been failing for a few days with the output of it's ext4 image being >2.0GB. This compresses it with bz2, thus solving the problem.
I'll add that currently we compress all these large images with bz2, which is very slow, I'd like to suggest a change to zstd or lzo. This would be specially meaningful in architectures where usually there are more cores at the expense of clock (since compression/decompression is single threaded.) In particular ARM families would benefit, with no downside to other arches. I am preparing this in another branch, but would like to see this one merged first.
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nix-review --run "nix-review wip"
./result/bin/
)nix path-info -S
before and after)Notify maintainers
cc @samueldr @andir @grahamc @cleverca22