nixos: make systemd-nspawn work out of the box on a NixOS root filesystem #35364

florianjacob · 2018-02-22T21:13:11Z

instead of creating an absolute symlink, which violates the os-release format
at https://www.freedesktop.org/software/systemd/man/os-release.html and
breaks systemd-nspawn and os-prober.

Motivation for this change

Fixes #28833, related to #9884.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

florianjacob · 2018-02-22T21:44:03Z

Requesting reviews from @palmeida and @rhendric to see how good this works.

fpletz

Looks good if this fixes the issue. :)

florianjacob · 2018-02-23T09:25:17Z

@palmeida confirmed this works for os-prober, but for nspawn, this is only enough for filesystems that were already booted once. The containerTarball generation code needs to copy /etc/os-release as well.

The easiest way to achieve that would be to add something like:

contents = [ { source = etc."os-release"; target = "/etc/os-release"; } ];

in the code for the docker container at
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/profiles/docker-container.nix#L20
which is used for lxc-based containerTarball as well, or for lxc only at https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/lxc-container.nix .

Not sure what makes more sense, though, especially I have no idea whether you want /etc/os-release in a Docker image. Doing research on that matter now. @offlinehacker @dezgeg opinions?

florianjacob · 2018-02-27T13:08:45Z

As docker is not really about starting OSes but more like start anything and just give me the right executable, in contrast to lxc, I now think putting this in lxc is the right way.

I added a commit which should copy /etc/os-release to the containerTarball, but I can't test it, because on my system, nix-build nixpkgs/nixos/release-combined.nix -A nixos.containerTarball.x86_64-linux against my branch in nixpkgs/ does not do anything, strangely. 🤔 If anybody knows what's going on there or can test this themselves, please let me know!

florianjacob · 2018-03-22T00:23:26Z

Found out the reason for why nix-build did just return nothing was actually in my code! I fixed that now! 🎉 for which I had to put /etc/os-release in the docker image as well, though. But as already said, should be fine, docker just ignores it and it's more like something informational to the admin regarding what this thing in here, far from a standard linux root fs hierarchy before the first boot, actually is.

The minimal commands to test this:

create /etc/systemd/nspawn/nixos-test.nspawn with

[Exec]
Boot=no
Parameters=/init

Execute:

$ nix-build path/to/nixpkgs/nixos/release.nix -A containerTarball.x86_64-linux
$ sudo mkdir -p /var/lib/machines/nixos-test
$ tar -C /var/lib/machines/nixos-test -xf /displayed/nix/store/path/…-tarball/tarball/nixos-system-x86_64-linux.tar.gz
$ sudo machinectl start nixos-test
$ sudo machinectl shell nixos-test
$ sudo machinectl stop nixos-test

Finally, just works, with no container tarball modifications necessary! As soon as hydra builds this thing, a machinectl pull-tar will suffice, without any image modifications! 😅

florianjacob · 2018-03-23T11:07:43Z

@rhendric reported problems with the Boot=no workaround for the docker profile creating /init and not /sbin/init (which I can't reproduce and therefore did not notice). Options:

a) make docker use /sbin/init as well, which would require changes in the docker files

b) make systemd-nspawn search at /init as well, if we want to avoid /sbin/init

c) split the config for docker and lxc, and fix this TODO while we are at it. I don't know what @edolstra had in mind there, though - move profiles/docker-container.nix to virtualization? Don't treat it as a separate file at all, but include it in the importing files virtualization/docker-image.nix and virtualization/lxc-container.nix?

Just discovered this, might be relevant as well: https://github.com/florianjacob/nixpkgs/blob/6a6847f15486eeba8e2b54d2710f0f61836b8cfe/nixos/modules/system/boot/loader/init-script/init-script.nix

dezgeg · 2018-03-23T15:47:20Z

I think making /sbin/init is a good idea given that lxc also looks for that by default, so it avoids one manual configuration step.

florianjacob · 2018-03-23T23:31:18Z

@dezgeg oh that is good to know, from a quick glance I thought lxc had no default places where it automatically checks for an init script, and only nspawn would profit. 👍

Ekleog · 2018-10-12T18:44:04Z

(triage) @florianjacob, are you still working on this?

florianjacob · 2018-10-15T22:53:07Z

@Ekleog Thanks, I forgot about this. It is essentially finished, I'm rebasing stuff now and applying the /sbin/init change to docker.

florianjacob · 2018-10-16T00:01:58Z

I now introduced the /init -> /sbin/init move.
This is ready for reviewers again, now no workaround at all is needed to run images with systemd-nspawn, as shown in the newly-added release notes.

As far as I know, this only is a breaking change for docker for people using docker to boot a full-blown NixOS instead of directly specifying the process they want to run.

edolstra · 2018-10-16T09:11:24Z

Not really in favor of replacing /etc symlinks by copies since it reduces the atomicity of the upgrade process. Maybe symlinks in /etc could be made relative (e.g. to ../nix/store)?

florianjacob · 2018-10-16T12:01:19Z

I'd prefer relative symlinks as well, and tried that route first, even without realizing the implications for the upgrade process. I found no way to create relative symlinks though within NixOS, and therefore used the existing ways to get a copy.

@edolstra you're suggesting to make all symlinks in /etc relative, if I understand you correctly?
That would solve /etc/os-release problem as well, and also make the other files in /etc easily readable when looking at the filesystem from outside (parallel installation, live cd, …)
If no one can think of problems with having relative symlinks in etc, I'll adjust nixos/modules/system/etc/make-etc.sh to count the slashes in $target and prepend that number + 1 of ../ to $source or something like that.

edolstra · 2018-10-16T13:52:19Z

@florianjacob Yes, sounds good!

so that the links can be followed if the NixOS installation is not mounted as filesystem root. In particular, this makes /etc/os-release adhere to the standard: https://www.freedesktop.org/software/systemd/man/os-release.html Fixes NixOS#28833.

out of the box by adding /etc/os-release to the image and putting init at the /sbin/init standard location.

florianjacob · 2018-11-13T09:47:45Z

I now switched everything to relative symlinks, mainly by using the ln -s --relative option, by which I could avoid brittle shell trickery. With this I could drop all copies, and it's indeed quite handy to be able to check out root filesystems without them being mounted at root. Everything else still seems to work, and upgrade was flawlessly.

This is now ready again for review and intensive testing - is there something like @GrahamcOfBorg test *? That would be quite handy. :D

danbst

I've rebased this PR on master and built on my system. So far so good. Then I checked for my main usecase, running nixos containers on non-nixos, and it failed. Left a comment. Here is patch:

From 9c0f4945fc96a14fe5ca4cc18bdbcb0b2a4d2b3e Mon Sep 17 00:00:00 2001
From: danbst <abcz2.uprola@gmail.com>
Date: Wed, 23 Jan 2019 22:19:50 +0200
Subject: [PATCH] make back /etc/static absolute symlink

---
 nixos/modules/system/etc/setup-etc.pl | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/nixos/modules/system/etc/setup-etc.pl b/nixos/modules/system/etc/setup-etc.pl
index 6cbf0e17793..82ef49a2a27 100644
--- a/nixos/modules/system/etc/setup-etc.pl
+++ b/nixos/modules/system/etc/setup-etc.pl
@@ -13,18 +13,26 @@ sub atomicSymlink {
     my ($source, $target) = @_;
     my $tmp = "$target.tmp";
     unlink $tmp;
-    # Create relative symlinks, so that the links can be followed if
-    # the NixOS installation is not mounted as filesystem root.
-    # Absolute symlinks violate the os-release format
-    # at https://www.freedesktop.org/software/systemd/man/os-release.html
-    # and break e.g. systemd-nspawn and os-prober.
+    symlink $source, $tmp or return 0;
+    rename $tmp, $target or return 0;
+    return 1;
+}
+
+# Create relative symlinks, so that the links can be followed if
+# the NixOS installation is not mounted as filesystem root.
+# Absolute symlinks violate the os-release format
+# at https://www.freedesktop.org/software/systemd/man/os-release.html
+# and break e.g. systemd-nspawn and os-prober.
+sub atomicRelativeSymlink {
+    my ($source, $target) = @_;
+    my $tmp = "$target.tmp";
+    unlink $tmp;
     my $rel = File::Spec->abs2rel($source, dirname $target);
     symlink $rel, $tmp or return 0;
     rename $tmp, $target or return 0;
     return 1;
 }
 
-
 # Atomically update /etc/static to point at the etc files of the
 # current configuration.
 atomicSymlink $etc, $static or die;
@@ -37,8 +45,7 @@ sub isStatic {
 
     if (-l $path) {
         my $target = readlink $path;
-        my $rel = File::Spec->abs2rel("/etc/static", dirname $path);
-        return substr($target, 0, length $rel) eq $rel;
+        return substr($target, 0, length "/etc/static/") eq "/etc/static/";
     }
 
     if (-d $path) {
@@ -111,7 +118,7 @@ sub link {
     if (-e "$_.mode") {
         my $mode = read_file("$_.mode"); chomp $mode;
         if ($mode eq "direct-symlink") {
-            atomicSymlink readlink("$static/$fn"), $target or warn;
+            atomicRelativeSymlink readlink("$static/$fn"), $target or warn;
         } else {
             my $uid = read_file("$_.uid"); chomp $uid;
             my $gid = read_file("$_.gid"); chomp $gid;
@@ -125,7 +132,7 @@ sub link {
         push @copied, $fn;
         print CLEAN "$fn\n";
     } elsif (-l "$_") {
-        atomicSymlink "$static/$fn", $target or warn;
+        atomicRelativeSymlink "$static/$fn", $target or warn;
     }
 }
 
-- 
2.19.2

Let me know if you'd like to polish this PR further, otherwise I'll grab your commit (that which doesn't touch docker) as a separate PR.

danbst · 2019-01-23T20:39:49Z

nixos/modules/system/etc/setup-etc.pl

+    # at https://www.freedesktop.org/software/systemd/man/os-release.html
+    # and break e.g. systemd-nspawn and os-prober.
+    my $rel = File::Spec->abs2rel($source, dirname $target);
+    symlink $rel, $tmp or return 0;


This is wrong. This works for system

$ ls -ld /etc/static lrwxrwxrwx 1 root root 53 січ 23 21:27 /etc/static -> ../nix/store/23r7mdcbqh5h7rfvdzifcygjfp9f5kyd-etc/etc

But fails for container:

# ls -ld /var/lib/containers/db2/etc/static lrwxrwxrwx 1 root root 53 Jan 23 15:07 /var/lib/containers/db2/etc/static -> ../nix/store/c7xn0a16907d2mi3kyidina9hg3r9nh4-etc/etc

That symlink doesn't resolve for container, when viewed from host

danbst · 2019-01-23T20:49:27Z

With my patch applied on top of this, I get following for containers:

# ls -l /var/lib/containers/db2/etc/static
lrwxrwxrwx 1 root root 51 Jan 23 15:29 /var/lib/containers/db2/etc/static -> /nix/store/c7xn0a16907d2mi3kyidina9hg3r9nh4-etc/etc
# ls -l /var/lib/containers/db2/etc/static/os-release
lrwxrwxrwx 1 root root 53 Dec 31  1969 /var/lib/containers/db2/etc/static/os-release -> ../../kxh43f5rrcj3620rl82jsmvxs95g5qwg-etc-os-release

Last one looks strange, but actually all good, it is a relative static/os-release symlink and resolves against /nix/store/c7xn0a16907d2mi3kyidina9hg3r9nh4-etc/etc

florianjacob · 2019-01-26T18:47:12Z

Then I checked for my main usecase, running nixos containers on non-nixos, and it failed.

That is really strange, as this is/was my main use case as well and I'm quite sure I tested that, as host acces to /etc/os-release is the main thing this PR is about 🤔

With my patch applied on top of this, I get following for containers:

# ls -l /var/lib/containers/db2/etc/static
lrwxrwxrwx 1 root root 51 Jan 23 15:29 /var/lib/containers/db2/etc/static -> > /nix/store/c7xn0a16907d2mi3kyidina9hg3r9nh4-etc/etc

If that's copied correctly then I'd argue that the first one looks odd, as it seems absolute. Will that really go to the container's nix store and not the host's nix store?

Let me know if you'd like to polish this PR further, otherwise I'll grab your commit (that which doesn't touch docker) as a separate PR.

Feel free to extract and handle that part if you're not using LXC for whatever you're doing. 👍
While my main use case has vanished as I've switched all my machines to NixOS in the meantime, I'm still interested in this work not getting lost.

danbst · 2019-01-26T19:22:57Z

That is really strange, as this is/was my main use case as well and I'm quite sure I tested that, as host acces to /etc/os-release is the main thing this PR is about

very strange.

If that's copied correctly then I'd argue that the first one looks odd, as it seems absolute. Will that really go to the container's nix store and not the host's nix store?

/var/lib/containers/db2/etc/static is absolute, all other links in /var/lib/containers/db2/etc are relative to /var/lib/containers/db2/etc/static.

How can /var/lib/containers/db2/etc/static be relative? Relative to what?

florianjacob · 2019-01-26T21:55:18Z

@danbst the main question is, what happens when you do cat /var/lib/containers/db2/etc/os-release on a host system where /nix/store does not exist, i.e. that neither NixOS nor has nix installed?

danbst · 2019-01-26T22:49:52Z

@florianjacob oh I see now. So in your case you distribute /nix/store with container, in my case I reuse /nix/store from the host.

Have to think about what is best solution for both cases

florianjacob · 2019-01-27T09:12:37Z

I'm happy I was able to bring my point across. 👍
I guess the /nix/store sharing is why this PR worked for me, but not for you.
The main idea here is to allow to get a full NixOS environment on non-NixOS distributions simply by running machinectl pull-tar https://hydra.org/…/containerTarball.x86_64-linux NixOS & machinectl start NixOS, so the /nix/store is contained in the image.

danbst · 2019-03-15T01:13:59Z

Update: relative symlinks turned out to be bad idea, because it is not easy to verify that relative symlink is actually a symlink to static (correct reverse of File::Spec->abs2rel is nontrivial)

fpletz · 2019-09-25T09:27:58Z

As I understand this PR, the functionality will be properly implemented by #67232. This closing for now. Feel free to reopen if you disagree.

florianjacob · 2019-09-25T12:37:08Z

@fpletz Thanks, was not aware of that new effort.

danbst · 2019-10-17T12:31:38Z

@fpletz @florianjacob #67232 was reverted

florianjacob mentioned this pull request Feb 22, 2018

/etc/os-release is an absolute symlink but should be a relative one for e.g. systemd-nspawn #28833

Open

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Feb 22, 2018

florianjacob mentioned this pull request Feb 22, 2018

Installing NixOS in a container with a non-NixOS host is painful #9884

Closed

fpletz added this to the 18.03 milestone Feb 22, 2018

fpletz approved these changes Feb 22, 2018

View reviewed changes

florianjacob force-pushed the os-release_copy branch 2 times, most recently from 1f23793 to d477f65 Compare February 27, 2018 12:59

florianjacob force-pushed the os-release_copy branch 2 times, most recently from 943ff33 to 6a6847f Compare March 22, 2018 00:06

GrahamcOfBorg added the 8.has: documentation label Mar 22, 2018

matthewbauer modified the milestones: 18.03, 18.09 Oct 1, 2018

samueldr modified the milestones: 18.09, 19.03 Oct 6, 2018

florianjacob force-pushed the os-release_copy branch 3 times, most recently from 76bd765 to ca28a30 Compare October 15, 2018 22:52

florianjacob force-pushed the os-release_copy branch from ca28a30 to 6c9126c Compare October 15, 2018 23:51

florianjacob changed the title ~~nixos/version: Copy /etc/os-release from nix store~~ nixos: make systemd-nspawn work out of the box on a NixOS root filesystem Oct 16, 2018

florianjacob force-pushed the os-release_copy branch from b2c5bfe to d18d332 Compare November 12, 2018 18:45

florianjacob requested a review from infinisil as a code owner November 12, 2018 18:45

florianjacob force-pushed the os-release_copy branch from d18d332 to fb00a87 Compare November 13, 2018 09:23

florianjacob added 2 commits November 13, 2018 10:37

nixos/container: Make images systemd-nspawn compatible

ba45d51

out of the box by adding /etc/os-release to the image and putting init at the /sbin/init standard location.

florianjacob force-pushed the os-release_copy branch from fb00a87 to ba45d51 Compare November 13, 2018 09:37

danbst mentioned this pull request Jan 2, 2019

extra-container on non-NixOS erikarvstedt/extra-container#1

Closed

danbst suggested changes Jan 23, 2019

View reviewed changes

danbst mentioned this pull request Jan 31, 2019

nixos: make symlinks in /etc relative (except /etc/static) #54980

Merged

10 tasks

ck3d mentioned this pull request Aug 23, 2019

machinectl compliant NixOS installation #67232

Merged

10 tasks

matthewbauer modified the milestones: 19.03, 20.03 Aug 28, 2019

fpletz closed this Sep 25, 2019

florianjacob deleted the os-release_copy branch September 25, 2019 12:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos: make systemd-nspawn work out of the box on a NixOS root filesystem #35364

nixos: make systemd-nspawn work out of the box on a NixOS root filesystem #35364

florianjacob commented Feb 22, 2018 •

edited

florianjacob commented Feb 22, 2018

fpletz left a comment

florianjacob commented Feb 23, 2018

florianjacob commented Feb 27, 2018

florianjacob commented Mar 22, 2018 •

edited

florianjacob commented Mar 23, 2018

dezgeg commented Mar 23, 2018

florianjacob commented Mar 23, 2018

Ekleog commented Oct 12, 2018

florianjacob commented Oct 15, 2018

florianjacob commented Oct 16, 2018

edolstra commented Oct 16, 2018

florianjacob commented Oct 16, 2018

edolstra commented Oct 16, 2018

florianjacob commented Nov 13, 2018

danbst left a comment

danbst Jan 23, 2019

danbst commented Jan 23, 2019

florianjacob commented Jan 26, 2019

danbst commented Jan 26, 2019

florianjacob commented Jan 26, 2019

danbst commented Jan 26, 2019

florianjacob commented Jan 27, 2019

danbst commented Mar 15, 2019

fpletz commented Sep 25, 2019

florianjacob commented Sep 25, 2019

danbst commented Oct 17, 2019

nixos: make systemd-nspawn work out of the box on a NixOS root filesystem #35364

nixos: make systemd-nspawn work out of the box on a NixOS root filesystem #35364

Conversation

florianjacob commented Feb 22, 2018 • edited

Motivation for this change

Things done

florianjacob commented Feb 22, 2018

fpletz left a comment

Choose a reason for hiding this comment

florianjacob commented Feb 23, 2018

florianjacob commented Feb 27, 2018

florianjacob commented Mar 22, 2018 • edited

florianjacob commented Mar 23, 2018

dezgeg commented Mar 23, 2018

florianjacob commented Mar 23, 2018

Ekleog commented Oct 12, 2018

florianjacob commented Oct 15, 2018

florianjacob commented Oct 16, 2018

edolstra commented Oct 16, 2018

florianjacob commented Oct 16, 2018

edolstra commented Oct 16, 2018

florianjacob commented Nov 13, 2018

danbst left a comment

Choose a reason for hiding this comment

danbst Jan 23, 2019

Choose a reason for hiding this comment

danbst commented Jan 23, 2019

florianjacob commented Jan 26, 2019

danbst commented Jan 26, 2019

florianjacob commented Jan 26, 2019

danbst commented Jan 26, 2019

florianjacob commented Jan 27, 2019

danbst commented Mar 15, 2019

fpletz commented Sep 25, 2019

florianjacob commented Sep 25, 2019

danbst commented Oct 17, 2019

florianjacob commented Feb 22, 2018 •

edited

florianjacob commented Mar 22, 2018 •

edited