Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/modules/system/boot/networkd: enable socket activation #89064

Merged
merged 1 commit into from Jun 15, 2020

Conversation

andir
Copy link
Member

@andir andir commented May 28, 2020

Motivation for this change

With this change systemd-networkd has it's netlink socket created via a
systemd.socket unit. One might think that this doesn't make much sense
since networkd is just going to create it's own socket on startup
anyway. The difference here is that we have configuration-time control
over things like socket buffer sizes vs compile-time constants.

For larger setups where networkd has to create a lot of (soft) devices
the default buffer size of currently 128MB is not enough.

A good example is a machine with >100 soft interfaces (e.g., wireguard
tunnels, VLANs, …) that all have to be brought up during startup. The
receive buffer size will spike due to all the generated message from the
new interfaces. Eventually some of the message will be dropped since
there is not enough (permitted) buffer space available.

By having networkd start through / with a netlink socket created by
systemd we can configure the ReceiveBufferSize parameter in the socket
options without recompiling networkd.

Since the actual memory requirements depend on hardware, timing, exact
configurations etc. it isn't currently possible to infer a good default
from within the NixOS module system. Administrators are advised to
monitor the logs of systemd-networkd for rtnl: kernel receive buffer overrun spam and increase the memory as required.

Note: Increasing the ReceiveBufferSize doesn't allocate any memory. It
just increases the upper bound on the kernel side. The memory allocation
depends on the amount of messages that are queued on the kernel side of
the netlink socket.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)

@flokli
Copy link
Contributor

flokli commented May 28, 2020

This (and a bit more) has already been done in #88158. However, it has gone stale a bit.

I like the note entry. Let's close #88158 and incorporate the remaining bits from there into here.

Copy link
Contributor

@flokli flokli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I propose cherry-picking over the timesyncd and resolved commits from #88158.

nixos/doc/manual/release-notes/rl-2009.xml Outdated Show resolved Hide resolved
nixos/doc/manual/release-notes/rl-2009.xml Outdated Show resolved Hide resolved
@@ -1179,6 +1179,7 @@ in

systemd.additionalUpstreamSystemUnits = [
"systemd-networkd.service" "systemd-networkd-wait-online.service"
"systemd-networkd.socket"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please put each list item in its own newline and sort.

nixos/modules/system/boot/networkd.nix Outdated Show resolved Hide resolved
@arianvp
Copy link
Member

arianvp commented May 28, 2020

This PR unlike mine is missing the dbus parts. Would be good to cherry pick that too

@flokli
Copy link
Contributor

flokli commented May 29, 2020

This PR unlike mine is missing the dbus parts. Would be good to cherry pick that too

Yeah, I mentioned that in my review

@andir
Copy link
Member Author

andir commented Jun 1, 2020

I will happily address all the review comments but I would like you both to reconsider merging these two PRs into one.

They both solve different issues and thus should both be accepted. We should try to avoid scope creep and keep things separated as they make logical sense (not purely by looking at the Diff but looking at the actual reasoning).

In contrast to #88158 this issue is not about early boot netlink messages or DBUS activation. It does something similar but with a different goal.

@arianvp
Copy link
Member

arianvp commented Jun 1, 2020 via email

@arianvp arianvp self-requested a review June 1, 2020 10:08
@flokli
Copy link
Contributor

flokli commented Jun 1, 2020

Alright, I reopened #88158. Please address the other review comments, they're still valid.

@flokli
Copy link
Contributor

flokli commented Jun 13, 2020

@andir with #88158 merged, can you rebase your changes on top of that?

@flokli
Copy link
Contributor

flokli commented Jun 13, 2020

If I haven't missed anything, it seems like all the config options set here were already set in #88158, so all that's left should be the release notes, right?

@flokli
Copy link
Contributor

flokli commented Jun 14, 2020

I rebased this on top of latest master.

Indeed the same config changes were already done in #88158 - your PR made networkd socket-activated to be able to change the socket buffer size, #88158 did it to buffer netlink messages in early boot - so this PR only contains release notes explaining the change itself, and some guidelines on when an increase of the buffer size is needed.

With this release <literal>systemd-networkd</literal> (when enabled through <xref linkend="opt-networking.useNetworkd"/>)
has it's netlink socket created through a <literal>systemd.socket</literal> unit. This gives us control over
socket buffer sizes and other parameters. For larger setups where networkd has to create a lot of (virtual)
devices the default buffer size (currently 128MB) is not enough.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were these 128MB also the buffer size when networkd wasn't socket-activated by systemd?

Since cd1deda systemd-networkd has it's
netlink socket created via a systemd.socket unit. One might think that
this doesn't make much sense since networkd is just going to create it's
own socket on startup anyway. The difference here is that we have
configuration-time control over things like socket buffer sizes vs
compile-time constants.

For larger setups where networkd has to create a lot of (virtual)
devices the default buffer size of currently 128MB is not enough.

A good example is a machine with >100 virtual interfaces (e.g.,
wireguard tunnels, VLANs, …) that all have to be brought up during
startup. The receive buffer size will spike due to all the generated
message from the new interfaces. Eventually some of the message will be
dropped since there is not enough (permitted) buffer space available.

By having networkd start through / with a netlink socket created by
systemd we can configure the `ReceiveBufferSize` parameter in the socket
options without recompiling networkd.

Since the actual memory requirements depend on hardware, timing, exact
configurations etc. it isn't currently possible to infer a good default
from within the NixOS module system. Administrators are advised to
monitor the logs of systemd-networkd for `rtnl: kernel receive buffer
overrun` spam and increase the memory as required.

Note: Increasing the ReceiveBufferSize doesn't allocate any memory.  It
just increases the upper bound on the kernel side. The memory allocation
depends on the amount of messages that are queued on the kernel side of
the netlink socket.
@flokli
Copy link
Contributor

flokli commented Jun 15, 2020

Rebased on latest master.

@flokli flokli merged commit 550e47c into NixOS:master Jun 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants