nixos/systemd: install sysctl snippets #66482

flokli · 2019-08-11T14:40:07Z

systemd provides two sysctl snippets, 50-coredump.conf and
50-default.conf.

Apart from some loose reverse path filtering and source routing
filtering, this enables fq_codel as a packet scheduler to fight
bufferbloat, and configures systemd-coredump properly.

Let's start using these, like other distros already do for quite some
time, and remove those duplicate boot.kernel.sysctl options we
previously did set.

In the case of rp_filter (which systemd would set to 2 (loose)), make
our overrides to "1" more explicit.

sysctl.d(5) recommends prefixing all filenames in /etc/sysctl.d with a
two-digit number and a dash, to simplify the ordering of the files.

Some packages provide custom files, often with "50-" prefix.
To ensure user-supplied configuration takes precedence over the one
specified via boot.kernel.sysctl, prefix the file generated there with
"60-".

Things done

Notify maintainers

cc @

flokli · 2019-08-11T14:44:42Z

Currently, the systemd test seems to break at another subtest ("file system with x-initrd.mount is not unmounted"), but the fq_codel one succeeds.

I didn't yet test whether coredumps are available via coredumpctl.

andir · 2019-08-11T14:55:37Z

nixos/modules/profiles/hardened.nix

  boot.kernel.sysctl."net.ipv4.conf.all.log_martians" = mkDefault true;
-  boot.kernel.sysctl."net.ipv4.conf.all.rp_filter" = mkDefault true;
+  boot.kernel.sysctl."net.ipv4.conf.all.rp_filter" = mkDefault "1";


Why do we have to change this? The sysctl module maps true to 1 for us (since toString true == "1").

I wondered the same. I think the answer is

In the case of rp_filter (which systemd would set to 2 (loose)), make
our overrides to "1" more explicit.

Yes. systemd sets these to "2" (loose) in the snippets already, the hardenened rules set it to "1" (strict).

Having a boolean here doesn't really explain what and why we're doing.
I also updated the comment above, to add the "strict" bit.

andir · 2019-08-11T15:18:48Z

OK, then it is reasonable IMO.

…

On Sun, 11 Aug 2019, 17:13 Joachim F., ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In nixos/modules/profiles/hardened.nix <#66482 (comment)>: > boot.kernel.sysctl."net.ipv4.conf.all.log_martians" = mkDefault true; - boot.kernel.sysctl."net.ipv4.conf.all.rp_filter" = mkDefault true; + boot.kernel.sysctl."net.ipv4.conf.all.rp_filter" = mkDefault "1"; I wondered the same. I think the answer is In the case of rp_filter (which systemd would set to 2 (loose)), make our overrides to "1" more explicit. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#66482?email_source=notifications&email_token=AAE365D472OZPAH6FBVN223QEAUAVA5CNFSM4IK36MJ2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCBGMVLQ#discussion_r312743279>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAE365ESEUYOM4EY67NES7TQEAUAVANCNFSM4IK36MJQ> .

nixos/doc/manual/release-notes/rl-1909.xml

grahamc · 2019-08-11T17:44:04Z

nixos/doc/manual/release-notes/rl-1909.xml

+     <para>
+       We now install the sysctl snippets shipped with systemd. Apart from some loose reverse path
+       filtering and source routing filtering, this enables <literal>fq_codel</literal> as a packet
+       scheduler to fight bufferbloat, and configures <literal>systemd-coredump</literal> properly.


I maybe should have left the above comment down a few lines. In general when it comes to release notes and setting changes, avoid weasel words.

About the rp_filter option from above comment: systemd sets net.ipv4.conf.all.rp_filter to 2 by default, which does enable some reverse path filtering, while we currently don't do any reverse path filtering at all. On hardened and google-compute-config, we set it to 1 (strict).

Detailled explanation about the options:

0 No source address validation is performed and any packet is forwarded to the destination network

1 Strict Mode as defined in RFC 3074. Each incoming packet to a router is tested against the routing table and if the interface that the packet is received on is not the best return path for the packet then the packet is dropped.

2 Loose mode as defines in RFC 3074 Loose Reverse Path. Each incoming packet is tested against the route table and the packet is dropped if the source address is not routable through any interface. The allows for asymmetric routing where the return path may not be the same as the source path.

As a lot of distros enabled the systemd sysctls, they enabled loose reverse path filtering, so I thought it should be fine to do so too. Same goes for net.ipv4.conf.all.accept_source_route = 0.

I didn't want to explicitly list every option currently set in ${systemd}/example/sysctl.d/50-default.conf - only the fact that we now apply these options, plus some broad summary about what's changed, so people are aware when reading release notes.

flokli · 2019-08-11T18:37:05Z

Regarding systemd-coredump:

We seem to currently have a systemd.coredump module, which duplicates the kernel.core_pattern from the upstream systemd unit if systemd.coredump.enable is set to true, otherwise sets it back to core.

I was a bit baffled coredumpctl doesn't save coredumps on NixOS by default, and wonder if it's time to merge the things enabled there to the systemd module itself (and remove the module) - people could still get the old behaviour by setting boot.kernel.sysctl."kernel.core_pattern" = "core".

@nh2, @Jookia @auntieNeo, what do you think?

nh2 · 2019-08-17T19:21:48Z

As long as we keep the ability to use normal core instead of systemd's coredump handling (because it's very slow slow), I'm fine with things being moved around.

flokli · 2019-08-17T19:30:27Z

@nh2 Yes, if you set the sysctl parameter as described in the changelog, systemd-coredump won't be handling coredumps at all, and the kernel will dump the coredump file by itself.

flokli · 2019-08-17T19:32:20Z

I'd be curious to know about when the coredump performance becomes critical though ;-)

flokli · 2019-08-17T20:40:01Z

I removed weaselness in the release notes and explained getting back the old coredump behaviour a bit more clearly in the release notes.

nh2 · 2019-08-17T21:12:19Z

I'd be curious to know about when the coredump performance becomes critical though ;-)

@flokli I have observed many times that when a program with a couple GB resident memory core dumps, systemd's coredump program was spinning on 100% CPU for many minutes, while core terminated almost instantly.

Probably it's this issue.

I didn't investigate it, I just found it very annoying to have to wait so long.

flokli · 2019-08-17T22:09:55Z

This was fixed upstream February 2016 in bdfd7b2c63cac944a3aa1fc0fafd19f41789e208 - anyways, when setting sysctl, it won't be piped through it.

nixos/doc/manual/release-notes/rl-1909.xml

worldofpeace · 2019-08-18T01:40:54Z

I found it impossible to suggest this, but my personal preference found this part to read pretty like this

diff --git a/nixos/doc/manual/release-notes/rl-1909.xml b/nixos/doc/manual/release-notes/rl-1909.xml
index f9363561bdc..3ae0492ebfa 100644
--- a/nixos/doc/manual/release-notes/rl-1909.xml
+++ b/nixos/doc/manual/release-notes/rl-1909.xml
@@ -435,13 +435,26 @@
    </listitem>
    <listitem>
      <para>
-       We now install the sysctl snippets shipped with systemd. Apart from enabling loose reverse path
-       filtering and source routing filtering, this enables <literal>fq_codel</literal> as a packet
-       scheduler to fight bufferbloat, and configures the kernel to pass coredumps to
-       <literal>systemd-coredump</literal>.
-       These sysctl snippets can be found in <literal>/etc/sysctl.d/50-*.conf</literal>,
-       and overridden via <link linkend="opt-boot.kernel.sysctl">boot.kernel.sysctl</link>
-       (which will place the parameters in <literal>/etc/sysctl.d/60-nixos.conf</literal>).
+       We now install the sysctl snippets that are shipped with systemd.
+       <itemizedlist>
+        <para>This enables:</para>
+        <listitem>
+          <para>Loose reverse path filtering</para>
+        </listitem>
+        <listitem>
+          <para>Source routing filtering</para>
+        </listitem>
+        <listitem>
+         <para>
+          <literal>fq_codel</literal> as a packet scheduler (this helps to fight bufferbloat)
+         </para>
+        </listitem>
+       </itemizedlist>
+
+       And configures the kernel to pass its coredumps to <literal>systemd-coredump</literal>.
+       The sysctl snippets that are shipped with systemd can be found in <literal>/etc/sysctl.d/50-*conf</literal> and can
+       be overridden with the <link linkend="opt-boot.kernel.sysctl">boot.kernel.sysctl</link> option.
+       Note these overrides still go into <literal>/etc/sysctl.d/60-nixos.conf</literal>.
      </para>
    </listitem>
    <listitem>

sysctl.d(5) recommends prefixing all filenames in /etc/sysctl.d with a two-digit number and a dash, to simplify the ordering of the files. Some packages provide custom files, often with "50-" prefix. To ensure user-supplied configuration takes precedence over the one specified via `boot.kernel.sysctl`, prefix the file generated there with "60-".

systemd provides two sysctl snippets, 50-coredump.conf and 50-default.conf. These enable: - Loose reverse path filtering - Source route filtering - `fq_codel` as a packet scheduler (this helps to fight bufferbloat) This also configures the kernel to pass coredumps to `systemd-coredump`. These sysctl snippets can be found in `/etc/sysctl.d/50-*.conf`, and overridden via `boot.kernel.sysctl` (which will place the parameters in `/etc/sysctl.d/60-nixos.conf`. Let's start using these, like other distros already do for quite some time, and remove those duplicate `boot.kernel.sysctl` options we previously did set. In the case of rp_filter (which systemd would set to 2 (loose)), make our overrides to "1" more explicit.

flokli · 2019-08-18T15:57:33Z

Docs should be good, any objections? ;-)

worldofpeace

Docs are good 👍

gebner · 2019-08-31T10:26:01Z

This change disables all SysRq keys (except sync) by default. Was this intentional? In any case, this needs to be documented in the release notes.

flokli · 2019-08-31T15:46:15Z

@gebner I indeed missed that line!

It's part of ${pkgs.systemd}/example/sysctl.d/50-default.conf:

# Use kernel.sysrq = 1 to allow all keys.
# See https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html for a list
# of values and keys.
kernel.sysrq = 16

I'll update the release notes accordingly and file a follow-up PR, thanks!

This was missing from NixOS#66482.

Since NixOS/nixpkgs#66482, it needs an explicit override.

flokli requested review from Mic92, fpletz and andir August 11, 2019 14:40

flokli requested a review from joachifm as a code owner August 11, 2019 14:40

ofborg bot added 6.topic: nixos 8.has: changelog 8.has: documentation 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Aug 11, 2019

andir reviewed Aug 11, 2019

View reviewed changes

grahamc reviewed Aug 11, 2019

View reviewed changes

nixos/doc/manual/release-notes/rl-1909.xml Outdated Show resolved Hide resolved

grahamc reviewed Aug 11, 2019

View reviewed changes

flokli mentioned this pull request Aug 11, 2019

🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 #66300

Closed

10 tasks

joachifm approved these changes Aug 15, 2019

View reviewed changes

flokli force-pushed the systemd-sysctl branch 2 times, most recently from dc7cf28 to 799b767 Compare August 16, 2019 09:32

flokli requested a review from infinisil as a code owner August 16, 2019 09:32

flokli force-pushed the systemd-sysctl branch from 799b767 to 8126744 Compare August 17, 2019 18:22

flokli force-pushed the systemd-sysctl branch 2 times, most recently from ccd422d to c7363bd Compare August 17, 2019 20:39

worldofpeace reviewed Aug 18, 2019

View reviewed changes

nixos/doc/manual/release-notes/rl-1909.xml Outdated Show resolved Hide resolved

worldofpeace reviewed Aug 18, 2019

View reviewed changes

nixos/doc/manual/release-notes/rl-1909.xml Outdated Show resolved Hide resolved

flokli force-pushed the systemd-sysctl branch 3 times, most recently from b42797b to 2f3e548 Compare August 18, 2019 15:50

flokli added 4 commits August 18, 2019 17:54

nixos/systemd: remove separate coredump module

bafc256

nixosTests.systemd: add fq_codel test

8e923df

flokli force-pushed the systemd-sysctl branch from 2f3e548 to 8e923df Compare August 18, 2019 15:54

worldofpeace approved these changes Aug 18, 2019

View reviewed changes

flokli merged commit 93a0317 into NixOS:master Aug 19, 2019

flokli deleted the systemd-sysctl branch August 19, 2019 14:32

aszlig mentioned this pull request Aug 30, 2019

nixos/tests/systemd: Fix x-initrd-mount flakiness #67798

Merged

flokli added a commit to flokli/nixpkgs that referenced this pull request Aug 31, 2019

release-notes: mention restricted SysRq key combinations

c48170a

This was missing from NixOS#66482.

flokli mentioned this pull request Aug 31, 2019

release-notes: mention restricted SysRq key combinations #67840

Merged

10 tasks

jtojnar mentioned this pull request Sep 13, 2019

It's not clear how to enable core dumps #60088

Closed

jtojnar added a commit to jtojnar/nixfiles that referenced this pull request Nov 5, 2019

kaiser: re-enable SysRq key

6f09a96

Since NixOS/nixpkgs#66482, it needs an explicit override.

deliciouslytyped mentioned this pull request Mar 29, 2020

Re-enable some SysRq-s disabled by pulling in systemd defaults #83694

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/systemd: install sysctl snippets #66482

nixos/systemd: install sysctl snippets #66482

flokli commented Aug 11, 2019

flokli commented Aug 11, 2019

andir Aug 11, 2019

joachifm Aug 11, 2019

flokli Aug 11, 2019

andir commented Aug 11, 2019 via email

grahamc Aug 11, 2019

flokli Aug 11, 2019

flokli commented Aug 11, 2019

nh2 commented Aug 17, 2019

flokli commented Aug 17, 2019

flokli commented Aug 17, 2019

flokli commented Aug 17, 2019

nh2 commented Aug 17, 2019

flokli commented Aug 17, 2019 •

edited

worldofpeace commented Aug 18, 2019 •

edited

flokli commented Aug 18, 2019

worldofpeace left a comment

gebner commented Aug 31, 2019

flokli commented Aug 31, 2019

nixos/systemd: install sysctl snippets #66482

nixos/systemd: install sysctl snippets #66482

Conversation

flokli commented Aug 11, 2019

Things done

Notify maintainers

flokli commented Aug 11, 2019

andir Aug 11, 2019

Choose a reason for hiding this comment

joachifm Aug 11, 2019

Choose a reason for hiding this comment

flokli Aug 11, 2019

Choose a reason for hiding this comment

andir commented Aug 11, 2019 via email

grahamc Aug 11, 2019

Choose a reason for hiding this comment

flokli Aug 11, 2019

Choose a reason for hiding this comment

flokli commented Aug 11, 2019

nh2 commented Aug 17, 2019

flokli commented Aug 17, 2019

flokli commented Aug 17, 2019

flokli commented Aug 17, 2019

nh2 commented Aug 17, 2019

flokli commented Aug 17, 2019 • edited

worldofpeace commented Aug 18, 2019 • edited

flokli commented Aug 18, 2019

worldofpeace left a comment

Choose a reason for hiding this comment

gebner commented Aug 31, 2019

flokli commented Aug 31, 2019

flokli commented Aug 17, 2019 •

edited

worldofpeace commented Aug 18, 2019 •

edited