How to replace

CapabilityBoundingSet = "";
ProtectHome = true;
MemoryDenyWriteExecute = true;

to

CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" "CAP_SYS_RESOURCE" ];
ProtectHome = mkDefault true;
MemoryDenyWriteExecute = !(builtins.any (mod: (mod.allowMemoryWriteExecute or false)) pkgs.nginx.modules);

@Izorkin Since I use mkOverride 900, you can just set sandbox = 2 and overwrite all changes you don't need in your module by just assigning as you did before.

FWIW, I think the top tier could use splitting up; ProtectSystem, ProtectHome etc. are useful drop-in hardening options for network-exposed services, but PrivateUsers, PrivateNetwork, CapabilityBoundingSet = [] and so on prevent it from being useful for the majority of services. Unfortunately, PrivateUsers prevents giving any additional host-level privileges (such as binding low ports) to the process at all; I'm not sure if this is an inherent limitation or just an infelicity of the current systemd implementation, but it prevented me from using it in #83474.

I support some kind of abstraction for hardening because I don't think individually setting 10+ options on each systemd service is going to scale, especially as new hardening options are introduced over time, and the status quo is very bad:

emily@renko ~> systemd-analyze security
UNIT                                 EXPOSURE PREDICATE HAPPY
ModemManager.service                      5.8 MEDIUM    😐    
NetworkManager-dispatcher.service         9.6 UNSAFE    😨    
NetworkManager.service                    7.8 EXPOSED   🙁    
accounts-daemon.service                   5.4 MEDIUM    😐    
avahi-daemon.service                      9.6 UNSAFE    😨    
bluetooth.service                         6.8 MEDIUM    😐    
colord.service                            8.8 EXPOSED   🙁    
dbus.service                              9.6 UNSAFE    😨    
display-manager.service                   9.8 UNSAFE    😨    
dnscrypt-proxy2.service                   8.3 EXPOSED   🙁    
emergency.service                         9.5 UNSAFE    😨    
geoclue.service                           7.4 MEDIUM    😐    
getty@tty1.service                        9.6 UNSAFE    😨    
getty@tty7.service                        9.6 UNSAFE    😨    
iwd.service                               5.9 MEDIUM    😐    
nix-daemon.service                        9.6 UNSAFE    😨    
nscd.service                              8.2 EXPOSED   🙁    
pcscd.service                             9.6 UNSAFE    😨    
polkit.service                            9.6 UNSAFE    😨    
rescue.service                            9.5 UNSAFE    😨    
rtkit-daemon.service                      7.1 MEDIUM    😐    
systemd-ask-password-console.service      9.3 UNSAFE    😨    
systemd-ask-password-wall.service         9.4 UNSAFE    😨    
systemd-initctl.service                   9.3 UNSAFE    😨    
systemd-journald.service                  4.4 OK        🙂    
systemd-logind.service                    2.8 OK        🙂    
systemd-machined.service                  6.1 MEDIUM    😐    
systemd-timesyncd.service                 2.1 OK        🙂    
systemd-udevd.service                     7.1 MEDIUM    😐    
udisks2.service                           9.6 UNSAFE    😨    
upower.service                            2.3 OK        🙂    
user@1000.service                         9.4 UNSAFE    😨    
zfs-zed.service                           9.6 UNSAFE    😨    
zpool-trim.service                        9.6 UNSAFE    😨    

emily@patchouli ~> systemd-analyze security
UNIT                                 EXPOSURE PREDICATE HAPPY
acme-dns.service                          1.5 OK        🙂    
dbus.service                              9.6 UNSAFE    😨    
dnscrypt-proxy2.service                   8.3 EXPOSED   🙁    
emergency.service                         9.5 UNSAFE    😨    
getty@tty1.service                        9.6 UNSAFE    😨    
nginx.service                             9.3 UNSAFE    😨    
nix-daemon.service                        9.6 UNSAFE    😨    
nscd.service                              8.2 EXPOSED   🙁    
rescue.service                            9.5 UNSAFE    😨    
sshd.service                              9.6 UNSAFE    😨    
systemd-ask-password-console.service      9.3 UNSAFE    😨    
systemd-ask-password-wall.service         9.4 UNSAFE    😨    
systemd-coredump@0.service                3.2 OK        🙂    
systemd-initctl.service                   9.3 UNSAFE    😨    
systemd-journald.service                  4.4 OK        🙂    
systemd-logind.service                    2.8 OK        🙂    
systemd-rfkill.service                    9.3 UNSAFE    😨    
systemd-timesyncd.service                 2.1 OK        🙂    
systemd-udevd.service                     7.1 MEDIUM    😐    
user@1000.service                         9.4 UNSAFE    😨    
zfs-zed.service                           9.6 UNSAFE    😨    
zpool-trim.service                        9.6 UNSAFE    😨    

But I do think it needs some careful consideration as to exactly what interface to expose. I think that rather than monotonic levels of sandboxing, several discrete named categories of sandboxing, plus some convenience bundles that contain common combinations, would be the best approach. Of course, some thought needs to be put in to avoid a proliferation of categories that don't reduce the complexity over the raw systemd options, but e.g. "protect the kernel stuff and devices", "reduce syscall attack surface", and "isolate filesystem with just the things I need" are viable categories that I think would reduce the complexity in hardening significantly.

Some options you haven't included that #83474 mentions / systemd-analyze security looks at: UMask, ProtectClock, ProtectKernelLogs, RestrictNamespaces.

By the way, it would be good to integrate the existing systemd-confinement.nix functionality, although it could use a rewrite to not use RootDirectory and avoid writeable mounts (see #64405 for some progress on that, though DynamicUser itself undoes a lot of mount sandboxing options so you unfortunately probably don't actually want to use it for hardened services in its current state).

An alternative to this module would be to set some hardening flags by default and only unset them in the services that actually need them. This will also remove the over all bloat of flags we need to set in every module.

@Mic92 this is actually what this module does. It does mkOverride 900 so services can override the sandboxing, for example Docker could use

{
  sandbox = 2;
  serviceConfig.ProtectControlGroups = false;
}

I also think sandbox = 2 is a bit optimistic and won't play nice with other security mechanisms we have (ProtectSystem and DynamicUser don't play well with systemd.services._name_.confinement); I'd rather see something more integrated if possible...

For example PrivateDevice=yes will mount a new /dev within the service chroot, but with confinement set to full it will actually lower the service security because there is now a /dev that didn't exist before! this stuff really isn't obvious to compose. And yes it's not reflected in systemd-analyse security as they don't know about what we've done.
So maybe we can do something more well thought out with the existing confinement module e.g. extend confinement.mode to have intermediate levels maybe?

For some other missing options though (mostly sandbox 1, ProtectHostname or RestrictRealtime really make sense to have always set by default... Even RestrictAddressFamilies...), I like the idea but I think it will be tricky.. Maybe a 3-steps process, like:

• add option
• let services use it and set workarounds if required
• make option default (conditioned on stateVersion)
  I'm not too sure how nixos deals with this kind of changes usually, just setting it by default based on stateVersion might work?
  But just adding the option isn't enough for me, many system modules that could benefit from it will never use it.

cc @aszlig for further opinion on this, I'm just a user of his superb confinement module :)

@Mic92 this is actually what this module does. It does mkOverride 900 so services can override the sandboxing, for example Docker could use

{
  sandbox = 2;
  serviceConfig.ProtectControlGroups = false;
}

I mean we could enable some sane default options all systemd services without having to enable sandbox explicitly.

I think this is a great idea, but imo there should only be two possibilities for the option:

• No sandboxing at all (0). This is insecure but the current default. Users can manually set systemd sandboxing options (aka blacklisting functionality) to restrict it
• Full sandboxing ("full"), making the sandboxing options as strict as systemd allows. Users will have to whitelist the functionality the service needs.

I think we should only use intermediate options for backwards compatibility. So in addition to 0 and "full" there would be 1, which is currently equivalent to "full". In the future a level 2 can be introduced, which would then become the new "full". So by setting "full" you opt into potentially having the service break in the future, requiring more whitelisting of options.

So currently the type should just be enum [ 0 1 "full" ]

Regarding turning on sandboxing by default, I think it should be done with these steps (with at least a NixOS release between them)

• Introduce this option, defaulting it to 0
• Add a warning when the option is not explicitly defined, in the lines of "systemd service XXX doesn't specify sandboxing level, defaulting to 0 (insecure). Set this option explicitly to silence this warning"
• Increase the default to the latest number, every time a new one is introduced

This makes the option essentially mandatory. If you choose to ignore the warning, the service might break in the future since the default changes. And if you don't ignore the warning and set the value, you either fix it to a backwards compatible number like 0 or 1, or you choose to intentionally make it backwards incompatible for the added security with "full"

I am very much a fan of sandboxing, but IMHO sandboxing needs to happen on a granular, and per-service level. Also see the discussion around #93305 (comment).

I don't think we should have a generic sandboxing module that hides/groups specific sandboxing properties behind a number, and potentially breaks services in the feature.
I don't think we can change the default of that option in any effective manner, or reason about what will change / might break when doing so.

"Not breaking existing services by being very explicit with what to sandbox" is also the route systemd went with each sandboxing option, for similar reasons.

IMHO, we should keep sandboxing and isolating more services, writing release notes on our way (so people know how to migrate / adapt), but I don't think a generic sandboxing module is the right approach.

@flokli

IMHO sandboxing needs to happen on a granular, and per-service level

This PR's module does not remove granularity and per-service config in any way. It merely is a way to conveniently set a lot of security-related settings to safer defaults. The user can still opt-out of each one and for each service separately.

I don't think we should have a generic sandboxing module that hides/groups specific sandboxing properties behind a number,

By combining these settings behind a number we make security easy: Instead of having to remember each and every one of the security settings you might want to enable, you can just set a single option which configures safe defaults. If the service doesn't work you can still opt-out of the settings that break it. This is a good thing, not a bad thing. In fact, this is kind of what NixOS is based on: Being able to just declare some enable option as true, influencing a whole bunch of stuff underneath, without the user having to worry about it.

and potentially breaks services in the feature.

Whether this setting breaks services in the future will be clearly specified, and it comes with reasonable tradeoffs:

• Either you set a specific number, which ensures that the service doesn't break in the future, but it might not use optimal security settings as they become available.
• Or you set "full", which ensures that the services uses all of systemd's available security settings, but that it might break with systemd updates. And this can easily be remediated with a test and/or regular maintenance.

I don't think we can change the default of that option in any effective manner, or reason about what will change / might break when doing so.

I agree that we may not be able to change the default of this option in the end, but this is a separate matter from this PR and doesn't need to influence the decision of merging it.

IMHO, we should keep sandboxing and isolating more services, writing release notes on our way (so people know how to migrate / adapt), but I don't think a generic sandboxing module is the right approach.

As seen from the current state of NixOS services, this obviously doesn't work. Most services don't use any security-related settings and are vastly under-restricted, systemd-analyze security is mostly red.

We can't expect module authors to know all systemd related security options and hope they enable them all as far as possible. Every time I write a service module I think to myself that while I could look into enabling all the security related settings, it would be a long and tedious process, and it would turn my 4-line service definition into a 30-line service definition, reducing readability and simplicity. So I don't do it.

Similarly, as a NixOS module reviewer, I could ask people to do the same, but who am I to tell people to do something I wouldn't even do myself. And what if they only add 3 security-related settings because they thought it was enough? Should I tell them to look into enabling all available ones? As a NixOS module reviewer, this just won't work well.

The alternative with this PR is to tell people to set sandbox = 1 and see if it works. If it does then great, you have a service that's reasonably secure. If not you'll get an error hinting at which more specific option needs to be enabled to make it work, which should lead to a working service after a couple iterations.

In essence, this is about being able to default to a secure set of settings for convenience.

@infinisil the most ideal spot for systemd hardening is upstream. I understand it is not realistic to expect every upstream project to accept all hardening instantly, but that doesn't mean we shouldn't work toward it.

When you take in mind the perspective of a software project that has been around for 30+ years we have to keep a couple things in mind:

• systemd is still an infant
• software has been running perfectly fine for decades without systemd hardening before systemd came along
• systemd does a ton of things that software used to have to do itself (or still has to do if it wants to run on BSD, for example) so systemd might not be a priority

Sure, we want to harden services, but we don't need to do it overnight with a switch. We need to carefully review (ourselves, and have upstream do the same) what we turn on so we stop breaking things without realizing we have done so.

@infinisil the most ideal spot for systemd hardening is upstream.

I think generally self-sandboxing is something more upstream projects should do regardless of systemd. For example if you restrict syscalls via seccomp BPF, the best way to do it would be the application itself, since you can even disallow things such as clone or execve, which is not easily possible via systemd (only if you use black magic™ and eg. use ptrace or LD_PRELOAD to inject the filter into the program).

The last point about breaking things is btw. exactly why I didn't start to add the the confinement module introduced a while ago to any NixOS modules yet, because it probably will break a lot of use cases I probably didn't think about because I don't use it in my systems. For example I usually run my PostgreSQL databases in separate network namespaces, which is fine since the listening socket is provided via systemd. However, this fails as soon as you start to use replication.

Personally, my approach is to set these settings on a per-service and per-machine basis for the services that I care about. For example I don't care about services like getty (which I usually disable anyway), the Nix daemon or any other service that's not exposed via network and confine all the services I do care about in such a way that they don't have access to anything else, including the services I don't care about :-)

Nevertheless, I do however still think that having a single option to restrict a service as much as possible is useful, maybe even on a global level, since whitelisting the features you actually care about is much easier than doing it the other way around.

My main concern with "this is too complicated for a single switch" is that it has the potential to result in perfection being the enemy of "good enough" in many cases.

As for the options themselves and what such a scale would look like I believe is useful research and exploration in itself.

Abandoning an idea because we know that it's likely an 80/20 solution to improve the status quo while waiting for upstream to eventually do all the "Right Things"™ feels premature to me. There is real value in exploring this space and I thought the PR was an excellent start to this conversation.

@infinisil the most ideal spot for systemd hardening is upstream. I understand it is not realistic to expect every upstream project to accept all hardening instantly, but that doesn't mean we shouldn't work toward it.

Yes agreed, hardening from upstream is nice. But this doesn't conflict with this PR at all. Just because upstream doesn't harden doesn't mean we can still do so in NixOS.

• software has been running perfectly fine for decades without systemd hardening before systemd came along

Of course software runs fine if they have all the privileges, but that's not really a good thing. But in any case, this also doesn't conflict with this PR. Unless "full" is used, services will still continue working. These sandbox settings are opt-in first and foremost, no defaults are changed.

Sure, we want to harden services, but we don't need to do it overnight with a switch. We need to carefully review (ourselves, and have upstream do the same) what we turn on so we stop breaking things without realizing we have done so.

Again, this PR won't break anything*. This only allows opt-in to secure defaults.

*: Actually one exception is when an upstream update needs new privileges. If the sandbox option is used, such a case might cause the update to break. But actually this is not such a bad thing, because upstream requiring new privileges should be looked at carefully.

Yes agreed, hardening from upstream is nice. But this doesn't conflict with this PR at all. Just because upstream doesn't harden doesn't mean we can still do so in NixOS.

This PR is problematic, as it provides a huge footgun. When enabled, it provides two handwavy sandboxing options (quoting from the description):

• 1: basic sandboxing, probably not breaking anything
• 2/full: the highest amount of sandboxing this module supports right now.

This is mostly tooling for other NixOS module authors. If you were to write a new module, you would probably pick 1, because you "want to have some sandboxing", but don't want to research the exact sandboxing mechanisms. You probably won't pick 2/full, because you don't want to see eventual breakage.

What would then happen is that more and more module authors would simply set 1, possibly adding some overrides if stuff doesn't work, without knowing what exactly is enabled.

In the future, when there's more sandboxing options, people will be afraid to touch the sandboxing module itself, and what is enabled by 1, because the description says "probably not breaking anything".
This effectively means we can't gradually introduce new sandboxing options without ensuring every service still works.

Again, I think sandboxing should be a conscious and fine-grained decision per service, ideally already done by upstream, or the downstream nixpkgs maintainers. I don't think a coarse "sandboxing shotgun" is the right way here. Also, hiding sandboxing options by an arbitrary number will make comparing with upstream systemd units, if they appear, much harder.

@flokli As I've argued in #87661 (comment), I don't think the current sandboxing levels of the PR should be used. I want levels

• 0 for no sandboxing, the default
• 1 for sandboxing of the first iteration of the module, this will have to be backwards compatible, we can't change any options this level sets
• full for the "latest" set of sandboxing options, currently just an alias to 1. Maybe this should be called latest instead. This is the variant, and the only one, that may break your service as new levels are introduced.

So there is a clear contract here:

• Set it to a number for backwards compatibility but not necessarily the latest set of defaults
• Or set it to full/latest for the latest set of defaults, but not necessarily backwards compatibility

And actually, we can even drop the full/latest level from this PR imo, and it would still be an improvement to the status quo. I want to be able to easily sandbox a module, and that is what this would achieve.

@infinisil then we still end up with module authors using 1 without clearly knowing what it does, and full, which few module authors will use, as it might suddenly break.

The only difference would be "we can't change the meaning of 1 anymore".

I still don't see the harm in duplicating some of the sandboxing directives in each module. In fact I'd argue having them easily diff-able with what upstream provides is an advantage :-)

People don't need to clearly know what 1 does, that's the entire point, to be able to just say "give me secure defaults", and be done, without worrying about what exactly those defaults are.

If people don't want to see a full setting that may break services over time, let's remove it for now. Such a PR is useful even without it.

And I already explained the problem with requiring people to add each setting manually in #87661 (comment).

From a quick grep through nixpkgs for serviceConfig, I'm estimating that around 85% of services don't use any security restrictions whatsoever, around 10% use a very small amount of restrictions, and around 5% use a lot of restrictions. Why do these 85% of services not use any restrictions? Because for one many people don't even know about them. And for another, that even if people know about them, they won't use them because it's a pain to set all of them. While we can of course try to educate people about all the settings they may set, I very much doubt this will have any noticeable impact on the problem.

While we can of course try to educate people about all the settings they may set, I very much doubt this will have any noticeable impact on the problem.

I'm still trying to understand why this is a burning problem. I don't see it as a problem, I see it as a room for growth in the GNU/Linux ecosystem 😄. Blind application of systemd sandboxing is one of the (many) problems veteran linux sysadmins see with systemd. Not saying they are right, not saying they are wrong... and I am saying I want to see us get there, but I want to see us get there "the right way", and without breaking too much stuff along the way.

I can imagine a future where systemd sandboxing isn't a mystery to the average upstream software developer. I would suppose hiding everything behind a 1 might not help get us there as fast as if people were to double down on working with upstream... 🤔

Oh, nice to see the conversation is back again. One thing to note is that this PR is a variation of the module we use in our downstream repository.
My idea was to make it more nixpkgiy by providing these sandbox levels. In our variant however we use these numbers differently.
Instead of denoting an amount of sandboxing where you choose the level you are interested in, they work like versions. When systemd adds more options, we can add more versions to the module and bump the sandboxing levels for each service on a one-by-one basis.
There are changes which shouldn't break too much but have a lot of side-effects (like ProtectClock breaking pppd).

The idea when we sandbox services downstream is to just set sandbox = 2 (our current latest level) and undo the changes that break the service (like PrivateUsers=false).
Some day (when I am more motivated), every service will be migrated to at least 2, so we can merge level 1 into 2, while also making it impossible for 1 to be used in the future by removing it from the enum.
Maybe this approach is more fesible for nixpkgs as well. Also our module does a bit more stuff like enabling our custom AppArmor module, while this makes it a lot less painful to sandbox stuff, it is probably out of scope for now.

@aanderse One of the problems (and also opportunities) nixpkgs has over other distros is that we mostly ship our own unit files instead of relying on random contrib/ units from upstream.
This allows us to roll out changes hardening the services, which may serve as a reference point for upstreaming these restrictions in the future ("We are running this for 2 years now, we doubt this will break for your users").
Also, if anything breaks for users, they are able to mkForce back the old values instead of having to wait for a backport or new release.
It's also good "PR" for nixpkgs to exceed the amount of security found on traditional systems which are using upstream units - it's a reason to give NixOS a try while also serving as a reference for other distros and upstream maintainers (like the AUR which can be used as a reference for "how to package X").

Oh also the main motiviation for the original module was ajs124 (not pinging here) who was sick of me copying the same chunks of text from my Joplin notebook.
This turned out to be quite annoying to review since he was never really sure which options were not copied because they had to be disabled. Also copy-pasting is just nasty when new options are added at a later point because one has to manually check all modules if these options are already set.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/hardening-systemd-services/17147/12