Calculate sha256 on the fly on dockerTools.buildLayeredImage #89075
Conversation
Previously, we were calculating layer checksums at a separate pass after the creation of the tarball. This is not ideal when packaging containers with large layers, since it requires reading the whole tarball again from disk in order to create the checksum. This commit calculates the sha256 in parallel when creating the tarball and saves it to a file for later use by buildLayeredImage function.
|
Please hold off on merging this since there seems to be a scary non-deterministic error somewhere. On every 2-3rd try, the tests fail with an error like: I couldn't figure out the reason yet, so I'd appreciate another set of eyes. |
|
|
Thank you @nlewo.
I think so. The error is not deterministic, so I can't be 100% sure, but I ran things without this patch multiple times and couldn't reproduce the error.
No, I just got the same error for layer 6. |
|
I built and loaded it 6 times and I can't reproduce locally: (I ran the gc before each attempt) Would you be able to keep a copy of each produced image in order to compare them? You could first try to run sha256 on each produced image to see if they are equal. If not, you could then use diffoscope to identity the differences (it know how to unpack Docker images). |
|
@nlewo thank you for taking a look! It is bit hard to reproduce, so I couldn't find enough time. But I am trying a different approach now, will update this PR in a week or so. |
|
This can be closed when #91084 is merged. |
|
#91084 got merged, closing. |
Motivation for this change
Previously, we were calculating layer checksums at a separate pass after
the creation of the tarball. This is not ideal when packaging containers
with large layers, since it requires reading the whole tarball again
from disk in order to create the checksum.
This commit calculates the sha256 in parallel when creating the tarball
and saves it to a file for later use by buildLayeredImage function.
Things done
Relevant NixOS tests (
nixosTests.docker-tools) pass.sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)