nixos/dockerPreloader: preload docker images #49379

nlewo · 2018-10-29T09:26:23Z

This module permits to preload Docker image in a VM in order to reduce
OIs on file copies. This module has to be only used in testing environments,
when the test requires several Docker images such as in Kubernetes
tests. In this case, virtualisation.dockerPreloader.images can
replace the services.kubernetes.kubelet.seedDockerImages options.

The idea is to populate the /var/lib/docker directory by mounting QCOW
files (we use QCOW files to avoid permission issues) that contain images.

For each image specified in
config.virtualisation.dockerPreloader.images:

The image is loaded by Docker in a VM
The resulting /var/lib/docker is written to a QCOW file

This set of QCOW files can then be used to populate the
/var/lib/docker:

Each QCOW is mounted in the VM
Symlink are created from these mount points to /var/lib/docker
A /var/lib/docker/image/overlay2/repositories.json file is generated
The docker daemon is started.

Motivation for this change

Speed up tests that require Docker images.

I have several Kubernetes tests that take more than 3min just to load Docker images (seedDockerImages k8s options). With this module, once QCOWs have been created, Docker images are immediately available in the test VMs (no more Docker load).

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

nlewo · 2018-10-29T09:41:57Z

@GrahamcOfBorg test docker-preloader

GrahamcOfBorg · 2018-10-29T09:42:09Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.docker-preloader

Partial log (click to expand)

Cannot nix-instantiate `tests.docker-preloader' because:
error: syntax error, unexpected INT, at /var/lib/gc-of-borg/nix-test-rs-24/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-24/nixos/release.nix:271:1

GrahamcOfBorg · 2018-10-29T09:48:23Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.docker-preloader

Partial log (click to expand)

Cannot nix-instantiate `tests.docker-preloader' because:
error: syntax error, unexpected INT, at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/nixos/release.nix:271:1

nlewo · 2018-10-29T09:51:39Z

@GrahamcOfBorg test docker-preloader

GrahamcOfBorg · 2018-10-29T09:51:49Z

Success on aarch64-linux

Attempted: tests.docker-preloader

No partial log is available.

GrahamcOfBorg · 2018-10-29T10:35:35Z

Success on x86_64-linux (full log)

Attempted: tests.docker-preloader

Partial log (click to expand)

docker: exit status 1
syncing
docker: running command: sync
docker: exit status 0
test script finished in 36.01s
cleaning up
killing docker (pid 597)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/a175r0302l5l3gb9wwyvppj1a039bk1x-vm-test-run-docker-preloader

This module permits to preload Docker image in a VM in order to reduce OIs on file copies. This module has to be only used in testing environments, when the test requires several Docker images such as in Kubernetes tests. In this case, `virtualisation.dockerPreloader.images` can replace the `services.kubernetes.kubelet.seedDockerImages` options. The idea is to populate the /var/lib/docker directory by mounting qcow files (we uses qcow file to avoid permission issues) that contain images. For each image specified in config.virtualisation.dockerPreloader.images: 1. The image is loaded by Docker in a VM 2. The resulting /var/lib/docker is written to a QCOW file This set of QCOW files can then be used to populate the /var/lib/docker: 1. Each QCOW is mounted in the VM 2. Symlink are created from these mount points to /var/lib/docker 3. A /var/lib/docker/image/overlay2/repositories.json file is generated 4. The docker daemon is started.

nlewo · 2018-10-29T14:13:37Z

cc @srhb @johanot @offlinehacker

zimbatm

nice. It's adding a bit of complications but it will be useful in various scenarios.

zimbatm · 2018-10-31T10:30:09Z

nixos/modules/virtualisation/docker-preloader.nix

+      # If docker.storageDriver is null, Docker choose the storage
+      # driver. So, in this case, we cannot be sure overlay2 is used.
+      assertion = cfg.dockerPreloader.images == []
+        || cfg.docker.storageDriver == "overlay2"


why is "overlay2" not compatible with overlay2?

It is compatible. The assertion is true if the storageDriver is overlay2.

zimbatm · 2018-10-31T10:30:52Z

nixos/tests/docker-preloader.nix

+
+    $docker->waitForUnit("sockets.target");
+    $docker->succeed("docker run nix nix-store --version");
+    $docker->succeed("docker run bash bash --version");


are we guaranteed that the VM won't access the network to fetch the image?

Not really. I don't think it's possible to disable remote registries in the Docker configuration. But this is guaranteed thanks to our sandbox.

srhb

Useful!

This module permits to preload Docker image in a VM in order to reduce OIs on file copies. This module has to be only used in testing environments, when the test requires several Docker images such as in Kubernetes tests. In this case, `virtualisation.dockerPreloader.images` can replace the `services.kubernetes.kubelet.seedDockerImages` options. The idea is to populate the /var/lib/docker directory by mounting qcow files (we uses qcow file to avoid permission issues) that contain images. For each image specified in config.virtualisation.dockerPreloader.images: 1. The image is loaded by Docker in a VM 2. The resulting /var/lib/docker is written to a QCOW file This set of QCOW files can then be used to populate the /var/lib/docker: 1. Each QCOW is mounted in the VM 2. Symlink are created from these mount points to /var/lib/docker 3. A /var/lib/docker/image/overlay2/repositories.json file is generated 4. The docker daemon is started.

nlewo force-pushed the docker-preload-images branch 5 times, most recently from 4092e12 to 5582ec9 Compare October 29, 2018 09:41

GrahamcOfBorg added 6.topic: nixos 8.has: documentation 8.has: module (update) labels Oct 29, 2018

nlewo force-pushed the docker-preload-images branch from 5582ec9 to cc66b18 Compare October 29, 2018 09:51

nlewo force-pushed the docker-preload-images branch from cc66b18 to 1f98898 Compare October 29, 2018 13:26

nlewo force-pushed the docker-preload-images branch from 1f98898 to 21bd8c7 Compare October 29, 2018 13:57

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 29, 2018

zimbatm approved these changes Oct 31, 2018

View reviewed changes

srhb approved these changes Oct 31, 2018

View reviewed changes

zimbatm merged commit 3fb4eb1 into NixOS:master Nov 3, 2018

DavHau mentioned this pull request Jan 22, 2020

docker-containers: add missing kubelet dependency for container systemd services #78313

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/dockerPreloader: preload docker images #49379

nixos/dockerPreloader: preload docker images #49379

nlewo commented Oct 29, 2018 •

edited

nlewo commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

nlewo commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

nlewo commented Oct 29, 2018

zimbatm left a comment

zimbatm Oct 31, 2018

nlewo Oct 31, 2018

zimbatm Oct 31, 2018

nlewo Oct 31, 2018

srhb left a comment

nixos/dockerPreloader: preload docker images #49379

nixos/dockerPreloader: preload docker images #49379

Conversation

nlewo commented Oct 29, 2018 • edited

Motivation for this change

Things done

nlewo commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

nlewo commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

nlewo commented Oct 29, 2018

zimbatm left a comment

Choose a reason for hiding this comment

zimbatm Oct 31, 2018

Choose a reason for hiding this comment

nlewo Oct 31, 2018

Choose a reason for hiding this comment

zimbatm Oct 31, 2018

Choose a reason for hiding this comment

nlewo Oct 31, 2018

Choose a reason for hiding this comment

srhb left a comment

Choose a reason for hiding this comment

nlewo commented Oct 29, 2018 •

edited