Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

squashfsTools: incorporate 4k-alignment option patch #49200

Merged
merged 1 commit into from Nov 17, 2018
Merged

squashfsTools: incorporate 4k-alignment option patch #49200

merged 1 commit into from Nov 17, 2018

Conversation

charles-dyfis-net
Copy link
Contributor

Motivation for this change

Padding the space between files in squashfs images increases the size of that image, but can greatly improve efficiency both of block- or extent-based on-disk deduplication (such as bees) and chunking-based software distribution systems (such as casync or desync).

This incorporates a patch from Chromium adding an option to enable such alignment, trivially tweaked so it applies to our tree. See https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/612449 for upstream discussion of said patch.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@Mic92
Copy link
Member

Mic92 commented Oct 27, 2018 •
edited

The discussion is from chromium's developers and not the authors of squashfs-tools, right?
In the thread they also discuss upstreaming those patches: https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/612449#message-659a92d5af13ebfe2c04b82cde01fb251e21cc2b

Have you tried sending the patch upstream? I would be interested in their opinion.

@charles-dyfis-net
Copy link
Contributor Author

I've requested upstream's feedback, and will follow up here when a reply is received (or they may comment directly -- a link to the PR was included).

@charles-dyfis-net
Copy link
Contributor Author

charles-dyfis-net commented Nov 16, 2018 •
edited

Silence from the upstream author, both directly and via mailing list (only message in the archive for October 2018, per https://sourceforge.net/p/squashfs/mailman/squashfs-devel/?viewmonth=201810 -- and as of this writing there's no content for November either).

I'm almost tempted to create a fork of the project that follows Google's publicly released patchset.

@Mic92
Copy link
Member

Mic92 commented Nov 16, 2018

@ruuda any opinions about the inclusion of the patch?

@Mic92
Copy link
Member

Mic92 commented Nov 16, 2018

There seems also no activity since 2014.

Mic92
Mic92 reviewed Nov 16, 2018
View reviewed changes
pkgs/tools/filesystems/squashfs/default.nix
@@ -22,6 +22,7 @@ stdenv.mkDerivation rec {
./0001-If-SOURCE_DATE_EPOCH-is-set-override-timestamps-with.patch
./0002-If-SOURCE_DATE_EPOCH-is-set-also-clamp-content-times.patch
./0003-remove-frag-deflator-thread.patch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move the comment from above directly to this patches and make a newline after them so it is clear that the 4k align patch is not meant for reproducibility?

@ruuda
Copy link
Contributor

ruuda commented Nov 16, 2018

I saw this patch before, seems like a useful thing to have.

I'm almost tempted to create a fork of the project that follows Google's publicly released patchset.

There is also the squashfskit fork which has seen more activity, perhaps they would accept it.

@Mic92
Copy link
Member

Mic92 commented Nov 16, 2018

Seems like a good idea, they also have patches we have: https://github.com/squashfskit/squashfskit/commits/master

@charles-dyfis-net
Copy link
Contributor Author

f6ec508 incorporates the improvement suggested by @Mic92.

@Mic92
Copy link
Member

Mic92 commented Nov 17, 2018

@GrahamcOfBorg build mksquashfs

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mksquashfs

Partial log (click to expand)

Cannot nix-instantiate `mksquashfs' because:
error: attribute 'mksquashfs' in selection path 'mksquashfs' not found

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mksquashfs

Partial log (click to expand)

Cannot nix-instantiate `mksquashfs' because:
error: attribute 'mksquashfs' in selection path 'mksquashfs' not found

@GrahamcOfBorg
Copy link

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: mksquashfs

Partial log (click to expand)

Cannot nix-instantiate `mksquashfs' because:
error: attribute 'mksquashfs' in selection path 'mksquashfs' not found

@Mic92
Copy link
Member

Mic92 commented Nov 17, 2018

@GrahamcOfBorg build squashfsTools

@Mic92 Mic92 changed the title mksquashfs: incorporate 4k-alignment option patch squashfsTools: incorporate 4k-alignment option patch Nov 17, 2018
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: squashfsTools

Partial log (click to expand)

cp unsquashfs /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev/bin
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev
shrinking /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev/bin/unsquashfs
shrinking /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev/bin/mksquashfs
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev/bin
patching script interpreter paths in /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev
checking for references to /build in /nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev...
/nix/store/4vsrv7shf98k3gqw8krmx4nfhr9rjmcv-squashfs-4.4dev

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: squashfsTools

Partial log (click to expand)

installing
install flags: SHELL=/nix/store/n9hba031gjky8hpjgx9fnlaxhidyzxbz-bash-4.4-p23/bin/bash XZ_SUPPORT=1 INSTALL_DIR=\$\{out\}/bin install
mkdir -p /nix/store/h8j3xnqad7m7zw3qlw2vwx32djwwfygx-squashfs-4.4dev/bin
cp mksquashfs /nix/store/h8j3xnqad7m7zw3qlw2vwx32djwwfygx-squashfs-4.4dev/bin
cp unsquashfs /nix/store/h8j3xnqad7m7zw3qlw2vwx32djwwfygx-squashfs-4.4dev/bin
post-installation fixup
strip is /nix/store/g5r4apl0za012ffs6ladinwa5w0m1l3k-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/h8j3xnqad7m7zw3qlw2vwx32djwwfygx-squashfs-4.4dev/bin
patching script interpreter paths in /nix/store/h8j3xnqad7m7zw3qlw2vwx32djwwfygx-squashfs-4.4dev
/nix/store/h8j3xnqad7m7zw3qlw2vwx32djwwfygx-squashfs-4.4dev

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: squashfsTools

Partial log (click to expand)

cp unsquashfs /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev/bin
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev
shrinking /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev/bin/unsquashfs
shrinking /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev/bin/mksquashfs
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev/bin
patching script interpreter paths in /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev
checking for references to /build in /nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev...
/nix/store/g2hp4h6x0dd14pz5dr2yacy1x32gwm9z-squashfs-4.4dev

@Mic92 Mic92 merged commit 9eefb44 into NixOS:master Nov 17, 2018
@charles-dyfis-net charles-dyfis-net mentioned this pull request Jul 6, 2019
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mic92 Mic92 Mic92 left review comments

@ruuda ruuda ruuda approved these changes

Assignees
No one assigned
Labels
10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@charles-dyfis-net @Mic92 @ruuda @GrahamcOfBorg