nixos: enrich systemd.nspawn.* options #74316

tekeri · 2019-11-27T09:56:36Z

Motivation for this change

Add a practical systemd-nspawn configuration options.

Possibly resolves #51076 and #40367 .

Things done

Notify maintainers

cc @c0bw3b @volth @arianvp @Mic92

andir

It looks good. Could you make sure we have a test for this feature?

tekeri · 2019-11-28T02:55:06Z

I'm new to NixOS tests, so it takes a while.

nixos/modules/system/boot/systemd-nspawn.nix

tekeri · 2020-01-21T09:47:34Z

@andir test is ready

nixos/tests/systemd-nspawn-options.nix

nixos/modules/system/boot/systemd-nspawn.nix

tekeri · 2020-01-23T09:48:17Z

Thanks, I'll fix them.

Ma27

Thanks a lot for your work on this and your patience! I have found a few more notes, but apart from that, the change seems pretty fine. Will test it as soon as I have time to :)

nixos/modules/system/boot/systemd-nspawn.nix

nixos/tests/systemd-nspawn-options.nix

flokli · 2020-07-11T22:22:12Z

@tekeri this needs another rebase.

@Ma27 did you get a chance to test this in the meantime?

Ma27 · 2020-07-11T23:58:45Z

@flokli I'm sorry, no. But added it on my todolist for tomorrow 😅

tekeri · 2020-07-13T08:11:01Z

pushed, while github is down

Ma27 · 2020-07-13T21:15:50Z

nixos/modules/system/boot/systemd-nspawn.nix

+          default = "";
+          description = ''
+            Optional mount options. See
+            <link xlink:href="url">https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html#Mount%20Options</link>


shouldn't the href attribute contain a URL

could you tell me how to make a link to the 'Mount options' section in the manual?

Ma27 · 2020-07-13T21:19:52Z

nixos/modules/system/boot/systemd-nspawn.nix

+        systemd.services."systemd-nspawn@".serviceConfig.ExecStart = [
+          ""  # deliberately empty. signals systemd to override the ExecStart
+          # Only difference between upstream is that we do not pass the -U flag
+          "${config.systemd.package}/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth --settings=override --machine=%i"


@flokli do you know if the systemd bug which made this necessary is fixed? Also, I'd prefer to set disableUnprivilegedNspawn to true by default since it's been a valid workaround for us.

Ma27 · 2020-07-13T21:22:21Z

nixos/modules/system/boot/systemd-nspawn.nix

+            ]
+          );
+        default = [];
+        example = [ "/tmp" { path = "/var/tmp"; mountOption = ""; } ];


Please use literalExample for examples that contain a list or an attr-path for better formatting in the manual.

Ma27 · 2020-07-13T21:27:33Z

nixos/tests/systemd-nspawn-options.nix

+  };
+in
+
+import ./make-test-python.nix ({ pkgs, ...} : {


Do you know why nscd inside the container breaks:

machine # [ 13.949141] systemd-nspawn[786]: [ OK ] Reached target Basic System. machine # [ 13.953850] systemd-nspawn[786]: Starting Name Service Cache Daemon... machine # [ 13.960114] systemd-nspawn[786]: Starting resolvconf update... machine # [ 13.982618] systemd-nspawn[786]: [FAILED] Failed to start Name Service Cache Daemon. machine # [ 13.984544] systemd-nspawn[786]: See 'systemctl status nscd.service' for details. machine # [ 13.988963] systemd-nspawn[786]: [ OK ] Reached target Host and Network Name Lookups. machine # [ 13.992789] systemd-nspawn[786]: [ OK ] Reached target User and Group Name Lookups. machine # [ 14.002572] systemd-nspawn[786]: Starting Login Service... machine # [ 14.054966] systemd-nspawn[786]: [FAILED] Failed to start Login Service. machine # [ 14.057025] systemd-nspawn[786]: See 'systemctl status systemd-logind.service' for details. machine # [ 14.061889] systemd-nspawn[786]: [ OK ] Stopped Login Service. machine # [ 14.068607] systemd-nspawn[786]: Starting Login Service... machine # [ 14.094932] systemd-nspawn[786]: [ OK ] Stopped Name Service Cache Daemon. machine # [ 14.101640] systemd-nspawn[786]: Starting Name Service Cache Daemon... machine # [ 14.129406] systemd-nspawn[786]: [FAILED] Failed to start Name Service Cache Daemon. machine # [ 14.131745] systemd-nspawn[786]: See 'systemctl status nscd.service' for details. machine # [ 14.154355] systemd-nspawn[786]: [FAILED] Failed to start Login Service. machine # [ 14.156893] systemd-nspawn[786]: See 'systemctl status systemd-logind.service' for details. machine # [ 14.163836] systemd-nspawn[786]: [ OK ] Stopped Login Service. machine # [ 14.178079] systemd-nspawn[786]: Starting Login Service... machine # [ 14.243987] systemd-nspawn[786]: [ OK ] Stopped Name Service Cache Daemon. machine # [ 14.249399] systemd-nspawn[786]: Starting Name Service Cache Daemon... machine # [ 14.258114] systemd-nspawn[786]: [FAILED] Failed to start Login Service. machine # [ 14.261136] systemd-nspawn[786]: See 'systemctl status systemd-logind.service' for details.

Ma27 · 2020-07-13T21:29:41Z

nixos/tests/systemd-nspawn-options.nix

+      # assert "Overlay=+/usr::/usr" in conf
+
+      # Checks filesConfig, networkConfig
+      assert "ReadOnly=false" in conf


I don't really think that we need this. As soon as some upstream options change we have to fix this test accordingly. Those kind of assertions don't belong in an integration test IMHO.

This is actually an unit test to check whether the boolean value is rendered as intended. Also, these lines gives an idea of how the options works.
No problem, however, to remove those assertions.

Ma27 · 2020-07-13T21:30:38Z

nixos/tests/systemd-nspawn-options.nix

+          "touch /var/lib/machines/test-container/etc/os-release",
+          "machinectl start test-container",
+          "machinectl terminate test-container",
+      )


Can you please add a more detailed test to make sure that the machine actually works? Some ideas:

Wait at least for multi-user.target.

Check if networking is fine.

Check if bind-mounts and restrictions do work.

FYI: This changeset is intended to be used with nixops-container backend. NixOS/nixops#1226
I'm internally using it and It works over 1 year with networking and bind-mounts.

tekeri · 2020-10-28T06:36:04Z

I'll fix the conflict and add tests when I have spare time.

bqv · 2020-11-25T16:43:15Z

Interest

hmenke · 2020-12-30T19:33:23Z

I haven't reviewed this PR in detail but I'm strongly in favor of a refactor of systemd.nspawn. Right now it's barely usable. I had to add some boilerplate which is nowhere to be found in the documentation to make the container boot at all.

{
  # Boilerplate
  systemd.targets.machines.enable = true;
  systemd.services."systemd-nspawn@archlinux" = {
    enable = true;
    wantedBy = [ "machines.target" ];
  };

  # Actual container
  systemd.nspawn."archlinux" = {
    enable = true; # does this even do anything?
    execConfig = {
      Boot = true;
    };
  };
}

But even with these options, the container still refuses to shut down properly on reboot and I'm stuck at the dreaded

[    **] A stop job is running for systemd-n…@archlinux.service (7s / 1min 30s)

In the journal it leaves

Dec 30 19:21:48 nixos systemd[1]: Stopping systemd-nspawn@archlinux.service...
Dec 30 19:21:48 nixos systemd-nspawn[499]: archlinux login: Trying to halt container. Send SIGTERM again to trigger immed>
Dec 30 19:21:48 nixos systemd-nspawn[499]: systemd 245.5-2-arch running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPAR>
Dec 30 19:21:48 nixos systemd-nspawn[499]: Detected virtualization systemd-nspawn.
Dec 30 19:21:48 nixos systemd-nspawn[499]: Detected architecture x86-64.
Dec 30 19:23:18 nixos systemd[1]: systemd-nspawn@archlinux.service: State 'stop-sigterm' timed out. Killing.

arianvp · 2020-12-31T09:54:46Z

Im pretty sure the machines.target is enabled by default... I agree that Systemd-nspawn@.service should have a wantedBy=machines.target by default though if that isn't the case. To align it with upstream.

…

On Wed, 30 Dec 2020, 20:33 Henri Menke, ***@***.***> wrote: I haven't reviewed this PR in detail but I'm strongly in favor of a refactor of systemd.nspawn. Right now it's barely usable. I had to add some boilerplate which is nowhere to be found in the documentation to make the container boot at all. { # Boilerplate systemd.targets.machines.enable = true; ***@***.***" = { enable = true; wantedBy = [ "machines.target" ]; }; # Actual container systemd.nspawn."archlinux" = { enable = true; # does this even do anything? execConfig = { Boot = true; }; }; } But even with these options, the container still refuses to shut down properly on reboot and I'm stuck at the dreaded [ **] A stop job is running for ***@***.*** (7s / 1min 30s) In the journal it leaves Dec 30 19:21:48 nixos systemd[1]: Stopping ***@***.*** Dec 30 19:21:48 nixos systemd-nspawn[499]: archlinux login: Trying to halt container. Send SIGTERM again to trigger immed> Dec 30 19:21:48 nixos systemd-nspawn[499]: systemd 245.5-2-arch running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPAR> Dec 30 19:21:48 nixos systemd-nspawn[499]: Detected virtualization systemd-nspawn. Dec 30 19:21:48 nixos systemd-nspawn[499]: Detected architecture x86-64. Dec 30 19:23:18 nixos systemd[1]: ***@***.***: State 'stop-sigterm' timed out. Killing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#74316 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZNI6XEG7JM2R6NNYB6I3SXN6BDANCNFSM4JSEIS6Q> .

arianvp · 2020-12-31T12:57:56Z

Let me rephrase:

I think having systemd.nspawn.<name>.enable imply systemd.services.'systemd-nspawn@<name>'.enable makes sense. We should add that.

the explicit wantedBy is needed because NixOS doesn't handle the upstream [Install] sections in unit files

So we should also add that.

That you needed

 systemd.targets.machines.enable = true;

surprises me though. enable = true is the default for all systemd units.

hmenke · 2020-12-31T14:38:34Z

@arianvp Indeed, I just checked again and systemd.targets.machines.enable = true; is not necessary. However, there is another problem with the systemd integration of NixOS. By specifying

{
  systemd.services."systemd-nspawn@archlinux".enable = true;
}

NixOS does not instantiate the systemd-nspawn@.service unit but instead creates a completely new unit named systemd-nspawn@archlinux.service which of course doesn't inherit anything from the uninstantiated unit. So this way of enabling the container is clearly wrong. To see this better, simply compare and contrast the output of systemctl cat systemd-nspawn@archlinux.service with systemctl cat systemd-nspawn@.service (or have a look at https://github.com/systemd/systemd/blob/master/units/systemd-nspawn%40.service.in). This is quite problematic because now essential options for the proper working of the container are missing.

I've opened a new issue for that: #108054

tekeri · 2021-01-04T11:27:00Z

@hmenke I think systemd-nspawn@$machine.service is necessary for integration with machinectl start $machine.
That is currently implemented as a call of systemctl start systemd-nspawn@$machine.service and no way to override the behavior.

stale · 2021-07-04T03:53:35Z

I marked this as stale due to inactivity. → More info

ofborg bot added 6.topic: nixos 8.has: module (update) labels Nov 27, 2019

tekeri force-pushed the enrich-systemd-nspawn branch from 2bd368e to 3415eb0 Compare November 27, 2019 10:08

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Nov 27, 2019

Ma27 self-requested a review November 27, 2019 11:05

andir reviewed Nov 27, 2019

View reviewed changes

Ma27 reviewed Dec 14, 2019

View reviewed changes

nixos/modules/system/boot/systemd-nspawn.nix Outdated Show resolved Hide resolved

nixos/modules/system/boot/systemd-nspawn.nix Outdated Show resolved Hide resolved

tekeri force-pushed the enrich-systemd-nspawn branch from 3415eb0 to 08abb1d Compare December 27, 2019 06:43

ofborg bot added 2.status: merge conflict and removed 2.status: merge conflict labels Dec 27, 2019

tekeri force-pushed the enrich-systemd-nspawn branch from f1b0fa3 to 23ab76d Compare January 14, 2020 08:29

tekeri force-pushed the enrich-systemd-nspawn branch from 23ab76d to f425b77 Compare January 21, 2020 09:46

ofborg bot added 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 0 labels Jan 21, 2020

tekeri requested a review from Ma27 January 22, 2020 04:48

Ma27 reviewed Jan 22, 2020

View reviewed changes

nixos/tests/systemd-nspawn-options.nix Outdated Show resolved Hide resolved

nixos/tests/systemd-nspawn-options.nix Outdated Show resolved Hide resolved

nixos/modules/system/boot/systemd-nspawn.nix Outdated Show resolved Hide resolved

tekeri force-pushed the enrich-systemd-nspawn branch from f425b77 to 1007deb Compare February 6, 2020 09:43

tekeri requested a review from Ma27 February 17, 2020 07:00

Ma27 reviewed Mar 8, 2020

View reviewed changes

tekeri force-pushed the enrich-systemd-nspawn branch from 1007deb to 4a7819c Compare April 3, 2020 04:07

arianvp mentioned this pull request Jun 3, 2020

systemd-nspawn can not start nixos-on-nixos #40367

Open

flokli requested a review from Ma27 June 14, 2020 10:01

Ma27 self-assigned this Jul 12, 2020

nixos: enrich systemd.nspawn.* options

701dec4

Ma27 force-pushed the enrich-systemd-nspawn branch from 8d61e3d to 701dec4 Compare July 13, 2020 21:14

Ma27 reviewed Jul 13, 2020

View reviewed changes

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2021

wegank marked this pull request as draft March 20, 2024 15:08

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos: enrich systemd.nspawn.* options #74316

nixos: enrich systemd.nspawn.* options #74316

tekeri commented Nov 27, 2019 •

edited

andir left a comment

tekeri commented Nov 28, 2019

tekeri commented Jan 21, 2020

tekeri commented Jan 23, 2020

Ma27 left a comment

flokli commented Jul 11, 2020

Ma27 commented Jul 11, 2020

tekeri commented Jul 13, 2020

Ma27 Jul 13, 2020

tekeri Jul 28, 2020 •

edited

Ma27 Jul 13, 2020

Ma27 Jul 13, 2020

Ma27 Jul 13, 2020

Ma27 Jul 13, 2020

tekeri Jul 14, 2020

Ma27 Jul 13, 2020

tekeri Jul 28, 2020

tekeri commented Oct 28, 2020

bqv commented Nov 25, 2020

hmenke commented Dec 30, 2020

arianvp commented Dec 31, 2020 via email

arianvp commented Dec 31, 2020 •

edited

hmenke commented Dec 31, 2020 •

edited

tekeri commented Jan 4, 2021

stale bot commented Jul 4, 2021

nixos: enrich systemd.nspawn.* options #74316

Are you sure you want to change the base?

nixos: enrich systemd.nspawn.* options #74316

Conversation

tekeri commented Nov 27, 2019 • edited

Motivation for this change

Things done

Notify maintainers

andir left a comment

Choose a reason for hiding this comment

tekeri commented Nov 28, 2019

tekeri commented Jan 21, 2020

tekeri commented Jan 23, 2020

Ma27 left a comment

Choose a reason for hiding this comment

flokli commented Jul 11, 2020

Ma27 commented Jul 11, 2020

tekeri commented Jul 13, 2020

Choose a reason for hiding this comment

tekeri Jul 28, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekeri commented Oct 28, 2020

bqv commented Nov 25, 2020

hmenke commented Dec 30, 2020

arianvp commented Dec 31, 2020 via email

arianvp commented Dec 31, 2020 • edited

hmenke commented Dec 31, 2020 • edited

tekeri commented Jan 4, 2021

stale bot commented Jul 4, 2021

tekeri commented Nov 27, 2019 •

edited

tekeri Jul 28, 2020 •

edited

arianvp commented Dec 31, 2020 •

edited

hmenke commented Dec 31, 2020 •

edited