nixos/containers: custom network namespaces #71328

d-xo · 2019-10-18T10:46:10Z

Motivation for this change

This change provides a declarative interface for creating named network namespaces, and spawning nixos containers within them.

This is useful if you:

Want multiple containers to communicate over localhost ("sidecar" containers)
Want to modify the network namespace from outside of the container

I use this for the second purpose. I want to be able to move interfaces in and out of a container's network namespace as part of a namespace based whole internet wireguard VPN solution (see here).

Things done

nixos/modules/tasks/network-namespaces.nix

asymmetric

Just tested this and it works as inteded 🎉 🎆. Other than the few nits, LGTM.

One thing: ideally systemd would implement network namespace creation natively. Until that day, we have to resort to scripting.

nixos/modules/virtualisation/containers.nix

nixos/modules/tasks/network-namespaces.nix

nixos/tests/containers-network-namespace.nix

nixos/modules/virtualisation/containers.nix

d-xo · 2019-11-15T17:18:35Z

rebased onto master and addressed review comments, much appreciated @asymmetric 🙏 🙂

asymmetric

One additional comment.

asymmetric · 2019-11-24T21:21:39Z

nixos/modules/tasks/network-namespaces.nix

+in
+{
+  options = {
+    networking.namespaces = mkOption {


We could split this into its own PR actually. This would make the current PR pretty straightforward and uncontroversial, and the two options are anyway orthogonal.

WDYT?

nixos/modules/tasks/network-namespaces.nix

nixos/tests/containers-network-namespace.nix

flokli · 2019-11-25T00:11:37Z

I'm also not entirely sure about whether we should add this - this will delete all namespaces previously configured every time if you add a new one and do a rebuild switch.

WDYT about something like systemd/systemd#11710?

andir

I would be in favor of having this (one unit per NS) instead of waiting for a networkd implementation. Going with this PR will enable users with scripted networking the ability to use this (without handcrafting the units). Also being able to restart things that are being setup is nice. networkd doesn't have that support yet and probably wont for a while.

The main pain point I see is that we are providing users with a new foot gun. If you tear down a NS with your physical interfaces in it those will be gone until you reboot/do some unknown udev incarnations. Since we aren't really providing the tooling to move interfaces in the container that should e fine tho…

nixos/modules/tasks/network-namespaces.nix

d-xo · 2019-11-26T11:56:08Z

rebased on master and addressed PR comments. Big thanks to @flokli and @andir for the review 🙏 🙂.

I made the following changes:

create one systemd unit per network namespace
escaped network namespace name when passing it into the unit definition
ported test to python and expanded it to test creation of multiple namespaces

@flokli I had not noticed that my previous approach could result in the deletion of already created network namespaces, thanks a lot for pointing this out. I believe that the one systemd unit per netns approach suggested by @andir should fix this, do you agree?

Regarding systemd/systemd#11710, I would be very happy to use it if and when it lands, but I wanted something simple and declarative that I could use in my config today. I would also note that since 412f6a9 we have modules in master that take network namespace names as arguments, so I think having a nixos native way to create them is beneficial today.

andir · 2019-11-26T12:08:15Z

Regarding systemd/systemd#11710, I would be very happy to use it if and when it lands, but I wanted something simple and declarative that I could use in my config today. I would also note that since 412f6a9 we have modules in master that take network namespace names as arguments, so I think having a nixos native way to create them is beneficial today.

Now that you are raising that I come to question if we should perhaps modify the wireguard module to make use of this instead of it rolling it's own. That would be a much more consistent user experience.

Also we should think about ordering the systemd units accordingly and by changing the wireguard module to make us of this PRs functionality we can even provide an example (and test?) of how to properly interact with it.

d-xo · 2019-11-27T12:36:58Z

Now that you are raising that I come to question if we should perhaps modify the wireguard module to make use of this instead of it rolling it's own. That would be a much more consistent user experience.

It's a good point, but I feel that network namespaces are a pretty generic feature with more uses than just wireguard VPN's, so I would prefer to keep facilities for their creation separate from the wireguard module. @asymmetric you added the namespace support to the wireguard module, what do you think?

we should think about ordering the systemd units accordingly

I currently have the namespace units set to before = [ "network-pre.target" ], and the wireguard units are set to after = [ "network-online.target" ]. Looking at it again now, the current before condition seems excessive, and I think that before = [ "network-online.target" ] should be sufficient?

by changing the wireguard module to make us of this PRs functionality we can even provide an example (and test?) of how to properly interact with it.

Very good point. I have added a commit to this PR updating the wireguard namespace tests to use this module for namespace creation.

nixos/tests/containers-network-namespace.nix

nixos/modules/tasks/network-namespaces.nix

asymmetric · 2019-12-03T09:58:42Z

Now that you are raising that I come to question if we should perhaps modify the wireguard module to make use of this instead of it rolling it's own. That would be a much more consistent user experience.

It's a good point, but I feel that network namespaces are a pretty generic feature with more uses than just wireguard VPN's, so I would prefer to keep facilities for their creation separate from the wireguard module. @asymmetric you added the namespace support to the wireguard module, what do you think?

I'm not sure I know what we're talking about. The Wireguard module doesn't create namespaces, and the feature I've added only expects them to already exist, so this change seems pretty orthogonal.

If you mean in the Wiregurad namespaces tests, then sure, I think it's good to use the new attirube there.

I think this is a good interim solution until networkd has native support for namespace creation.

d-xo · 2019-12-10T16:17:19Z

rebased on master and addressed review comments.

I'm not sure I know what we're talking about. The Wireguard module doesn't create namespaces, and the feature I've added only expects them to already exist, so this change seems pretty orthogonal.

@asymmetric I had understood the question to be whether the wireguard module should start creating network namespaces itself. Although reading back I may have been getting a little confused :/

Allow declarative creation of named network namespaces

Allow nixos containers to be launched in a pre-existing named network namespace

d-xo · 2019-12-11T10:46:57Z

rebased on master and cleaned up the tests.

asymmetric · 2019-12-13T15:20:31Z

@asymmetric I had understood the question to be whether the wireguard module should start creating network namespaces itself.

@xwvvvvwx @andir I think the two concerns are pretty orthogonal, and by not tying them together too much we leave the option open for other ways of creating the namespace to emerge in the future. As long as it's documented (which it is), I think users should be tasked with creating the namespace however they see fit.

Also, if the namespace already exists, the wireguard module will then break, unless you make namespace creation idempotent, which seems pretty hacky.

All in all, I think we shouldn't do it. WDYT?

asymmetric · 2019-12-13T15:25:19Z

nixos/modules/tasks/network-namespaces.nix

+        Type = "oneshot";
+        RemainAfterExit = true;
+        ExecStart = "${pkgs.iproute}/bin/ip netns add ${strings.escapeShellArg netns}";
+        ExecStop = "${pkgs.iproute}/bin/ip netns del ${strings.escapeShellArg netns}";


According to the ip-netns manpage:

It is possible to lose the physical device when it was moved to netns and then this netns was deleted with a running process: $ ip netns add net0 $ ip link set dev eth0 netns net0 $ ip netns exec net0 SOME_PROCESS_IN_BACKGROUND $ ip netns del net0 and eth0 will appear in the default netns only after SOME_PROCESS_IN_BACKGROUND will exit or will be killed. To prevent this the processes running in net0 should be killed before deleting the netns: $ ip netns pids net0 | xargs kill $ ip netns del net0

Do you think it's necessary to document this in the module?

This is usually used to move wg* interfaces into a namespace, right?

So it's probably not too much of an issue. Adding a link to these docs sounds like a good idea, but we don't need to copy this paragraph in.

netvl · 2020-05-09T08:43:35Z

Hello,

Is there any chance for this to get merged? This seems like a very useful piece of functionality.

asymmetric · 2020-05-11T10:41:27Z

nixos/modules/tasks/network-namespaces.nix

+    value = {
+      description = "Network Namespace: ${netns}";
+      before = [ "network-pre.target" ];
+      wantedBy = [ "multi-user.target" ];


Nit: According to this documentation page, one should set Wants=network-pre.target.

I don't t know if there will be a difference in practice for most users, but I do think there's some dissonance in binding namespace creation to the multi-user.target.

OTOH the wireguard module also does WantedBy=multi-user.target, so maybe this is a convention that makes sense in NixOS?

Citing from above docs:

Services that want to be run before the network is configured should place Before=network-pre.target and also set Wants=network-pre.target to pull it in.

We should add Wants=network-pre.target.

I don't understand the question regarding multi-user.target.

errnoh · 2020-06-10T18:51:58Z

Started using this as well, everything works as advertised on 20.03. Wouldn't mind getting this functionality merged at some point.

asymmetric · 2020-06-11T14:30:07Z

I would like to see this land too, it's been way too long and it's already shown itself to be useful to several people.

One day networkd will support this usecase natively, we will remove scripted networking and everything will be simpler. But that day is not here yet, and I see no reason not to incrementally improve the status quo, when we can.

stale · 2021-06-04T23:55:59Z

I marked this as stale due to inactivity. → More info

asymmetric · 2021-06-05T10:11:54Z

Still relevant

stale · 2022-01-09T00:07:22Z

I marked this as stale due to inactivity. → More info

d-xo requested a review from infinisil as a code owner October 18, 2019 10:46

d-xo commented Oct 18, 2019

View reviewed changes

nixos/modules/tasks/network-namespaces.nix Show resolved Hide resolved

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 18, 2019

asymmetric reviewed Nov 6, 2019

View reviewed changes

asymmetric reviewed Nov 9, 2019

View reviewed changes

nixos/modules/virtualisation/containers.nix Outdated Show resolved Hide resolved

d-xo force-pushed the containers/custom-netns branch from 42ba32d to 897b108 Compare November 15, 2019 17:15

asymmetric reviewed Nov 24, 2019

View reviewed changes

flokli requested changes Nov 25, 2019

View reviewed changes

flokli requested review from fpletz and andir November 25, 2019 00:11

andir reviewed Nov 25, 2019

View reviewed changes

nixos/modules/tasks/network-namespaces.nix Outdated Show resolved Hide resolved

nixos/modules/tasks/network-namespaces.nix Outdated Show resolved Hide resolved

d-xo force-pushed the containers/custom-netns branch from 897b108 to d47dcd6 Compare November 26, 2019 11:56

d-xo force-pushed the containers/custom-netns branch from d47dcd6 to c3c9bda Compare November 27, 2019 12:36

flokli reviewed Dec 2, 2019

View reviewed changes

nixos/tests/containers-network-namespace.nix Outdated Show resolved Hide resolved

asymmetric reviewed Dec 3, 2019

View reviewed changes

nixos/modules/tasks/network-namespaces.nix Outdated Show resolved Hide resolved

d-xo force-pushed the containers/custom-netns branch from c3c9bda to 0f674d0 Compare December 10, 2019 16:13

d-xo added 2 commits December 11, 2019 11:21

nixos/networking: add networking.namespaces

18a3843

Allow declarative creation of named network namespaces

nixos/containers: allow spawning containers in an existing netns

b6d0a66

Allow nixos containers to be launched in a pre-existing named network namespace

d-xo force-pushed the containers/custom-netns branch from 0f674d0 to 77566a8 Compare December 11, 2019 10:22

tests: add test for containers.<name>.networkNamespace

2ea771c

tests: declarative netns creation in wireguard tests

d23591a

d-xo force-pushed the containers/custom-netns branch from 77566a8 to d23591a Compare December 11, 2019 10:37

d-xo mentioned this pull request Dec 11, 2019

nixos-containers: allow writable api fs #73467

Draft

10 tasks

asymmetric reviewed Dec 13, 2019

View reviewed changes

asymmetric reviewed May 11, 2020

View reviewed changes

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md and removed 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md labels Jun 4, 2021

asymmetric mentioned this pull request Nov 9, 2021

[RFC 0108] NixOS Container rewrite NixOS/rfcs#108

Merged

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 9, 2022

wegank marked this pull request as draft March 20, 2024 15:10

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/containers: custom network namespaces #71328

nixos/containers: custom network namespaces #71328

d-xo commented Oct 18, 2019 •

edited

asymmetric left a comment

d-xo commented Nov 15, 2019

asymmetric left a comment

asymmetric Nov 24, 2019

flokli commented Nov 25, 2019

andir left a comment

d-xo commented Nov 26, 2019 •

edited

andir commented Nov 26, 2019

d-xo commented Nov 27, 2019 •

edited

asymmetric commented Dec 3, 2019 •

edited

d-xo commented Dec 10, 2019 •

edited

d-xo commented Dec 11, 2019

asymmetric commented Dec 13, 2019

asymmetric Dec 13, 2019

flokli Jun 11, 2020

netvl commented May 9, 2020

asymmetric May 11, 2020

flokli Jun 11, 2020

errnoh commented Jun 10, 2020

asymmetric commented Jun 11, 2020 •

edited

stale bot commented Jun 4, 2021

asymmetric commented Jun 5, 2021

stale bot commented Jan 9, 2022

nixos/containers: custom network namespaces #71328

Are you sure you want to change the base?

nixos/containers: custom network namespaces #71328

Conversation

d-xo commented Oct 18, 2019 • edited

Motivation for this change

Things done

asymmetric left a comment

Choose a reason for hiding this comment

d-xo commented Nov 15, 2019

asymmetric left a comment

Choose a reason for hiding this comment

asymmetric Nov 24, 2019

Choose a reason for hiding this comment

flokli commented Nov 25, 2019

andir left a comment

Choose a reason for hiding this comment

d-xo commented Nov 26, 2019 • edited

andir commented Nov 26, 2019

d-xo commented Nov 27, 2019 • edited

asymmetric commented Dec 3, 2019 • edited

d-xo commented Dec 10, 2019 • edited

d-xo commented Dec 11, 2019

asymmetric commented Dec 13, 2019

asymmetric Dec 13, 2019

Choose a reason for hiding this comment

flokli Jun 11, 2020

Choose a reason for hiding this comment

netvl commented May 9, 2020

asymmetric May 11, 2020

Choose a reason for hiding this comment

flokli Jun 11, 2020

Choose a reason for hiding this comment

errnoh commented Jun 10, 2020

asymmetric commented Jun 11, 2020 • edited

stale bot commented Jun 4, 2021

asymmetric commented Jun 5, 2021

stale bot commented Jan 9, 2022

d-xo commented Oct 18, 2019 •

edited

d-xo commented Nov 26, 2019 •

edited

d-xo commented Nov 27, 2019 •

edited

asymmetric commented Dec 3, 2019 •

edited

d-xo commented Dec 10, 2019 •

edited

asymmetric commented Jun 11, 2020 •

edited