Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/nixpkgs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 5961f211f3bc
Choose a base ref
...
head repository: NixOS/nixpkgs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 0d983f9f99b4
Choose a head ref
  • 2 commits
  • 2 files changed
  • 2 contributors

Commits on Dec 30, 2019

  1. dockerTools.buildLayeredImage: fix building layered images in parallel 

    when tar'ing store paths into layered archives when building layered
images, don't use the absolute nix store path so that tar won't complain
if something new is added to the nix store

when building the final docker image, ignore any file changes tar
detects in the layers. they are all immutable and the only thing that
might change is the number of hard links due to store optimization
    Richard Wallace committed Dec 30, 2019
    Copy the full SHA
    3be7675 View commit details

Commits on Jan 11, 2020

  1. Merge pull request #75911 from Simspace/parallel-docker-buildlayeredi… 

    …mage

when building a layered docker image, ignore it if tar encounters cha…
    @nlewo
    nlewo authored Jan 11, 2020
    Copy the full SHA
    0d983f9 View commit details
Showing with 48 additions and 7 deletions.
  1. +16 −2 pkgs/build-support/docker/default.nix
  2. +32 −5 pkgs/build-support/docker/store-path-to-layer.sh
18 changes: 16 additions & 2 deletions pkgs/build-support/docker/default.nix
View file
Original file line number Diff line number Diff line change
@@ -325,7 +325,6 @@ rec {
| jshon -d config \
| jshon -s "1970-01-01T00:00:01Z" -i created > generic.json
# WARNING!
# The following code is fiddly w.r.t. ensuring every layer is
# created, and that no paths are missed. If you change the
@@ -625,7 +624,22 @@ rec {
-i "$imageName" > image/repositories
echo "Cooking the image..."
tar -C image --dereference --hard-dereference --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=0 --group=0 --mode=a-w --xform s:'^./':: -c . | pigz -nT > $out
# tar exits with an exit code of 1 if files changed while it was
# reading them. it considers a change in the number of hard links
# to be a "change", which can cause this to fail if images are being
# built concurrently and auto-optimise-store is turned on. since
# know the contents of these files will not change, we can reasonably
# ignore this exit code
set +e
tar -C image --dereference --hard-dereference --sort=name \
--mtime="@$SOURCE_DATE_EPOCH" --owner=0 --group=0 \
--mode=a-w --xform s:'^./':: --use-compress-program='pigz -nT' \
--warning=no-file-changed -cf $out .
RET=$?
if [ $RET -ne 0 ] && [ $RET -ne 1 ]; then
exit $RET
fi
set -e
echo "Finished."
'';
37 changes: 32 additions & 5 deletions pkgs/build-support/docker/store-path-to-layer.sh
View file
Original file line number Diff line number Diff line change
@@ -5,16 +5,43 @@ set -eu
layerNumber=$1
shift

storePath="$1"
shift

layerPath="./layers/$layerNumber"
echo "Creating layer #$layerNumber for $@"
echo "Creating layer #$layerNumber for $storePath"

mkdir -p "$layerPath"
tar --no-recursion -rf "$layerPath/layer.tar" \

# make sure /nix and /nix/store appear first in the archive.
# we create the directories here and use them because
# when there are other things being added to the
# nix store, tar could fail, saying,
# "tar: /nix/store: file changed as we read it"
mkdir -p nix/store
tar -cf "$layerPath/layer.tar" \
--mtime="@$SOURCE_DATE_EPOCH" \
--owner=0 --group=0 /nix /nix/store
tar -rpf "$layerPath/layer.tar" --hard-dereference --sort=name \
--owner=0 --group=0 \
--transform='s,nix,/nix,' \
nix

# we change into the /nix/store in order to avoid a similar
# "file changed as we read it" error as above. Namely,
# if we use the absolute path of /nix/store/123-pkg
# and something new it added to the nix store while tar
# is running, it will detect a change to /nix/store and
# fail. Instead, if we cd into the nix store and copy
# the relative nix store path, tar will ignore changes
# to /nix/store. In order to create the correct structure
# in the tar file, we transform the relative nix store
# path to the absolute store path
n=$(basename "$storePath")
tar -C /nix/store -rpf "$layerPath/layer.tar" \
--hard-dereference --sort=name \
--mtime="@$SOURCE_DATE_EPOCH" \
--owner=0 --group=0 "$@"
--owner=0 --group=0 \
--transform="s,$n,/nix/store/$n," \
$n

# Compute a checksum of the tarball.
tarhash=$(tarsum < $layerPath/layer.tar)