nixos/nginx: use sandboxing mode #85567

Izorkin · 2020-04-19T15:31:44Z

Motivation for this change

PR based on #60646
Enable run nginx web service in sandboxing mode.

Things done

dasJ · 2020-04-19T18:31:55Z

A fully up-to-date list of hardening options our sandbox module applies:

{
            # Filesystem stuff
            ProtectSystem = "strict"; # Prevent writing to most of /
            ProtectHome = true; # Prevent accessing /home and /root
            PrivateTmp = true; # Give an own directory under /tmp
            PrivateDevices = true; # Deny access to most of /dev
            ProtectKernelTunables = true; # Protect some parts of /sys
            ProtectControlGroups = true; # Remount cgroups read-only
            RestrictSUIDSGID = true; # Prevent creating SETUID/SETGID files
            PrivateMounts = true; # Give an own mount namespace

            # Capabilities
            CapabilityBoundingSet = ""; # Allow no capabilities at all
            NoNewPrivileges = true; # Disallow getting more capabilities. This is also implied by other options.

            # Kernel stuff
            ProtectKernelModules = true; # Prevent loading of kernel modules
            SystemCallArchitectures = "native"; # Usually no need to disable this
            ProtectKernelLogs = true; # Prevent access to kernel logs
            ProtectClock = true; # Prevent setting the RTC

            # Networking
            RestrictAddressFamilies = ""; # Example: "AF_UNIX AF_INET AF_INET6"
            PrivateNetwork = true; # Isolate the entire network

            # Misc
            LockPersonality = true; # Prevent change of the personality
            ProtectHostname = true; # Give an own UTS namespace
            RestrictRealtime = true; # Prevent switching to RT scheduling
            MemoryDenyWriteExecute = true; # Maybe disable this for interpreters like python
            PrivateUsers = true; # If anything randomly breaks, it's mostly because of this
}

I understand some of these are not useful at all (like PrivateNetwork), but maybe you find some of these useful.

Also, ProtectClock and ProtectKernelLogs are not yet supported by the NixOS systemd.

Izorkin · 2020-04-19T19:06:57Z

With PrivateUsers = true; error:

nginx.service: Failed to set up user namespacing: No space left on device
nginx.service: Failed at step USER spawning /nix/store/snsli9q204d8xhiwx8vr5faq5snd153c-unit-script-nginx-pre-start: No space left on device

With ProtectClock = true; and ProtectKernelLogs = true; warning:

systemd[1]: /nix/store/bsjqsa1jpy2kgxr5h0y177hcn7wpv9df-unit-nginx.service/nginx.service:26: Unknown key name 'ProtectClock' in section 'Service', ignoring.
systemd[1]: /nix/store/bsjqsa1jpy2kgxr5h0y177hcn7wpv9df-unit-nginx.service/nginx.service:30: Unknown key name 'ProtectKernelLogs' in section 'Service', ignoring.

With CapabilityBoundingSet = "";

nginx.service: Failed to apply ambient capabilities (before UID change): Operation not permitted
nginx.service: Failed at step CAPABILITIES spawning /nix/store/snsli9q204d8xhiwx8vr5faq5snd153c-unit-script-nginx-pre-start: Operation not permitted

dasJ · 2020-04-19T19:08:08Z

Our nginx module uses these for the capabilities:

      CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" "CAP_SYS_RESOURCE" ];
      AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" "CAP_SYS_RESOURCE" ];

Izorkin · 2020-04-19T19:26:32Z

Updated.

dasJ · 2020-04-19T19:42:57Z

Are you sure it needs AF_NETLINK? It's weird because it's lacking from our module, but is then allowed in AppArmor later on 🤔 @ajs124 any idea?

ajs124 · 2020-04-19T19:45:22Z

I have no recollection of anything related to this, so you might as well dig through the code yourself @dasJ.

dasJ · 2020-04-19T19:47:00Z

Also it's worth mentioning in the changelog that MemoryDenyWriteExecute will probably break the nginx lua module 😩

Izorkin · 2020-04-19T19:51:45Z

Remove this option

Also it's worth mentioning in the changelog that MemoryDenyWriteExecute will probably break the nginx lua module 😩

Remove the option MemoryDenyWriteExecute?

dasJ · 2020-04-19T19:56:30Z

It's a useful way of mitigating more severe problems when an attacker has already gained some access, so I'd keep it but people using the lua module should know that it breaks the module.

Izorkin · 2020-04-19T19:57:52Z

Add this wariant?
MemoryDenyWriteExecute = mkDefault true;

dasJ · 2020-04-19T19:58:51Z

Probably for the best, yes :)

Izorkin · 2020-04-19T20:01:46Z

With MemoryDenyWriteExecute = true; nginx not worked, time-out open test page.

Izorkin · 2020-04-19T20:14:10Z

Not worked with pkgs.nginxModules.pagespeed

Izorkin · 2020-04-19T20:48:49Z

@dasJ need backport to 20-03?

flokli · 2020-04-19T21:37:09Z

This can't be backported to 20.03, which is almost released already.

I'm also a bit afraid about the things some of these options might break - do any other distros already have a nginx unit with some sandboxing options we could take some inspiration from?

Ma27 · 2020-04-20T00:07:15Z

@GrahamcOfBorg build nginx
@GrahamcOfBorg test nginx

nixos/doc/manual/release-notes/rl-2009.xml

flokli · 2020-04-20T15:33:28Z

nixos/modules/services/web-servers/nginx/default.nix

+        CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" "CAP_SYS_RESOURCE" ];
+        # Security
+        NoNewPrivileges = true;
+        # Sandboxing
+        ProtectSystem = "strict";
+        ProtectHome = true;
+        PrivateTmp = true;
+        PrivateDevices = true;
+        ProtectHostname = true;
+        ProtectKernelTunables = true;
+        ProtectKernelModules = true;
+        ProtectControlGroups = true;
+        RestrictAddressFamilies = [ "AF_UNIX" "AF_INET" "AF_INET6" ];
+        LockPersonality = true;
+        MemoryDenyWriteExecute = mkDefault true;
+        RestrictRealtime = true;
+        RestrictSUIDSGID = true;
+        PrivateMounts = true;
+        # System Call Filtering
+        SystemCallArchitectures = "native";


Again, I'd feel wayy more comfortable if we'd have more tests covering standard usecases, and see they're not broken.

Need create test to check sandbox mode with perl /lua scripts?

yes. We need more tests to ensure things are not broken.

nixos/modules/services/web-servers/nginx/default.nix

Mic92 · 2020-05-12T14:19:26Z

nixos/modules/services/web-servers/nginx/default.nix

+      } // optionalAttrs cfg.enableSandbox {
+        # Sandboxing
+        ProtectSystem = "strict";
+        ProtectHome = mkDefault true;


How about moving ProtectSystem, ProtectHome and MemoryDenyWriteExecute to enableSandbox and move other options above the conditional? I think those 3 options are the controversial ones that could break with peoples setup, where we need more feedback.

This variant?

# Security NoNewPrivileges = true; # Sandboxing PrivateTmp = true; PrivateDevices = true; ProtectHostname = true; ProtectKernelTunables = true; ProtectKernelModules = true; ProtectControlGroups = true; RestrictAddressFamilies = [ "AF_UNIX" "AF_INET" "AF_INET6" ]; LockPersonality = true; RestrictRealtime = true; RestrictSUIDSGID = true; PrivateMounts = true; # System Call Filtering SystemCallArchitectures = "native"; } // optionalAttrs cfg.enableSandbox { ProtectSystem = "strict"; ProtectHome = mkDefault true; MemoryDenyWriteExecute = !(builtins.any (mod: (mod.allowMemoryWriteExecute or false)) pkgs.nginx.modules); };

Yes. This looks good to me.

In this variant, it will not work to disable the options PrivateTmp and etc.

Right. PrivateTmp should be also in the sandbox section. The rest looks safe to me, but correct me if I am wrong.

Or leave the current variant, or rename cfg.enableSandbox to cfg.enableSandboxStrict.

I prefer the current variant.

nixos/modules/services/web-servers/nginx/default.nix

nixbitcoin · 2020-05-12T14:43:43Z

I integrated the earlier module from #60646 in https://github.com/nixbitcoin/nix-bitcoin/tree/harden-nginx and never had any problems. I would be very happy to see services.nginx.enableSandbox implemented in nixpkgs.

nixbitcoin · 2020-05-12T14:46:46Z

Consider adding RemoveIPC systemd hardening option.

Izorkin · 2020-05-12T15:48:59Z

Added RemoveIPC

Izorkin · 2020-05-12T16:23:58Z

@GrahamcOfBorg test nginx
@GrahamcOfBorg test nginx-sandbox

nixos/modules/services/web-servers/nginx/default.nix

The allowMemoryWriteExecute option is required to checking enabled nginxModules and disable the nginx sandbox mode MemoryDenyWriteExecute.

(cherry picked from commit 6c437ef)

nh2 · 2020-11-08T15:52:38Z

nixos/doc/manual/release-notes/rl-2009.xml

+   <listitem>
+     <para>
+       Add option <literal>services.nginx.enableSandbox</literal> to starting Nginx web server with additional sandbox/hardening options.
+       By default, write access to <literal>services.nginx.stateDir</literal> is allowed. To allow writing to other folders,


I think that the release notes should also have mentioned that due to the added ProtectHome = mkDefault true;, nginx can no longer serve files from users' home directories.

It's not uncommon to serve subdirectories of home directories publicly (e.g. /home/*/public_html/), and this change breaks that without any mention.

See NixOS#85567 (review)

nh2 · 2020-11-08T16:10:00Z

I think that the release notes should also have mentioned that due to the added ProtectHome = mkDefault true;, nginx can no longer serve files from users' home directories.

I made PR #103147 for that.

Needs 20.09 backport.

…-notes manual: nginx: Mention ProtectHome in release notes. See #85567

See NixOS#85567 (review) (cherry picked from commit 2f845dc)

ofborg bot added 6.topic: nixos 8.has: changelog 8.has: documentation 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Apr 19, 2020

Izorkin force-pushed the nginx-sandbox branch from 995c1e8 to 8a0a2e6 Compare April 19, 2020 19:25

Izorkin force-pushed the nginx-sandbox branch from 8a0a2e6 to 0cd0b4a Compare April 19, 2020 19:54

Izorkin force-pushed the nginx-sandbox branch 2 times, most recently from 6a4b65f to 47f89ba Compare April 19, 2020 20:51

Ma27 requested review from globin and fpletz April 20, 2020 00:08

flokli requested changes Apr 20, 2020

View reviewed changes

dasJ mentioned this pull request May 12, 2020

nixos/systemd-sandbox: A generic sandboxing module #87661

Closed

10 tasks

Mic92 reviewed May 12, 2020

View reviewed changes

nixos/modules/services/web-servers/nginx/default.nix Show resolved Hide resolved

Mic92 reviewed May 12, 2020

View reviewed changes

nixbitcoin reviewed May 12, 2020

View reviewed changes

nixos/modules/services/web-servers/nginx/default.nix Show resolved Hide resolved

Izorkin force-pushed the nginx-sandbox branch 2 times, most recently from 9838d49 to fc58cb5 Compare May 12, 2020 15:48

Mic92 reviewed May 12, 2020

View reviewed changes

nixos/modules/services/web-servers/nginx/default.nix Outdated Show resolved Hide resolved

Izorkin added 6 commits May 12, 2020 20:03

nixos/nginx: enable sandboxing

628354c

nixos/nginx: add release notes

97a0928

nixos/tests: fix nginx-pubhtml test

af6d009

nixos/tests: add nginx-sandbox test

c710661

nginxModules: add option allowMemoryWriteExecute

aa12fb8

The allowMemoryWriteExecute option is required to checking enabled nginxModules and disable the nginx sandbox mode MemoryDenyWriteExecute.

nixos/nginx: add option enableSandbox

94391fc

Izorkin force-pushed the nginx-sandbox branch from fc58cb5 to 94391fc Compare May 12, 2020 17:03

Mic92 merged commit 6c437ef into NixOS:master May 13, 2020

Izorkin deleted the nginx-sandbox branch May 13, 2020 11:15

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request May 13, 2020

Merge pull request NixOS#85567 from Izorkin/nginx-sandbox

6b663ad

(cherry picked from commit 6c437ef)

aanderse mentioned this pull request Jul 6, 2020

transmission: apply RFC0042 and harden the service #92106

Merged

24 tasks

nh2 reviewed Nov 8, 2020

View reviewed changes

nh2 added a commit to nh2/nixpkgs that referenced this pull request Nov 8, 2020

manual: nginx: Mention ProtectHome in release notes. See NixOS#85567.

2f845dc

See NixOS#85567 (review)

nh2 mentioned this pull request Nov 8, 2020

manual: nginx: Mention ProtectHome in release notes. See #85567 #103147

Merged

nh2 added a commit that referenced this pull request Nov 8, 2020

Merge pull request #103147 from nh2/nginx-sandbox-protecthome-release…

1c460c0

…-notes manual: nginx: Mention ProtectHome in release notes. See #85567

nh2 added a commit to nh2/nixpkgs that referenced this pull request Nov 8, 2020

manual: nginx: Mention ProtectHome in release notes. See NixOS#85567.

8226a4d

See NixOS#85567 (review) (cherry picked from commit 2f845dc)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/nginx: use sandboxing mode #85567

nixos/nginx: use sandboxing mode #85567

Izorkin commented Apr 19, 2020 •

edited

dasJ commented Apr 19, 2020 •

edited

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020

ajs124 commented Apr 19, 2020

dasJ commented Apr 19, 2020

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020 via email

Izorkin commented Apr 19, 2020

Izorkin commented Apr 19, 2020

Izorkin commented Apr 19, 2020

flokli commented Apr 19, 2020

Ma27 commented Apr 20, 2020 •

edited

flokli Apr 20, 2020

Izorkin Apr 20, 2020

flokli Apr 21, 2020

Mic92 May 12, 2020

Izorkin May 12, 2020

Mic92 May 12, 2020

Izorkin May 12, 2020

Mic92 May 12, 2020 •

edited

Izorkin May 12, 2020

Izorkin May 12, 2020

nixbitcoin commented May 12, 2020

nixbitcoin commented May 12, 2020 •

edited

Izorkin commented May 12, 2020

Izorkin commented May 12, 2020

nh2 Nov 8, 2020

nh2 commented Nov 8, 2020

nixos/nginx: use sandboxing mode #85567

nixos/nginx: use sandboxing mode #85567

Conversation

Izorkin commented Apr 19, 2020 • edited

Motivation for this change

Things done

dasJ commented Apr 19, 2020 • edited

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020

ajs124 commented Apr 19, 2020

dasJ commented Apr 19, 2020

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020

Izorkin commented Apr 19, 2020

dasJ commented Apr 19, 2020 via email

Izorkin commented Apr 19, 2020

Izorkin commented Apr 19, 2020

Izorkin commented Apr 19, 2020

flokli commented Apr 19, 2020

Ma27 commented Apr 20, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 May 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixbitcoin commented May 12, 2020

nixbitcoin commented May 12, 2020 • edited

Izorkin commented May 12, 2020

Izorkin commented May 12, 2020

Choose a reason for hiding this comment

nh2 commented Nov 8, 2020

Izorkin commented Apr 19, 2020 •

edited

dasJ commented Apr 19, 2020 •

edited

Ma27 commented Apr 20, 2020 •

edited

Mic92 May 12, 2020 •

edited

nixbitcoin commented May 12, 2020 •

edited