Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/httpd: modernize module standards #85043

Merged
merged 4 commits into from Apr 26, 2020
Merged

Conversation

aanderse
Copy link
Member

@aanderse aanderse commented Apr 12, 2020

Motivation for this change
  • security
  • performance
  • promote modern standards

Under what scenario might the new defaults in this PR not be right for you?

  • if you use services.httpd.enablePHP - you should read upstream httpd documentation and carefully decide if you want to have services.httpd.mpm set to event and services.httpd.virtualHosts.<name?>.httpd2 set to true
  • if you use services.httpd.enablePerl - again, carefully read upstream httpd documentation and carefully decide if you want to have services.httpd.mpm set to event and services.httpd.virtualHosts.<name?>.httpd2 set to true
  • if you use services.httpd.virtualHosts.<name?>.enableUserDir and your permissions aren't loose enough you might want the httpd master process to run as root
  • if you use either services.httpd.enablePHP or services.httpd.enablePerl you should take special care with your ssl certificates as the same user executing your perl and php scripts will have at least read access to your ssl certificates

To put the above points into context... it is 2020 and you probably shouldn't be using enablePHP, enablePerl, enableUserDir, or setting services.httpd.mpm to prefork unless you are supporting legacy code which can't or won't be updated.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc @FPtje @immae @redvers

@Mic92
Copy link
Member

Mic92 commented Apr 12, 2020

It might be useful for users to document, what people should use instead of enablePhp and enablePerl, maybe as part of module documentation (meta.docs).

@Izorkin
Copy link
Contributor

Izorkin commented Apr 13, 2020

@aanderse planned add sandboxing?
Worked with this exapmple:

        # Security
        NoNewPrivileges = true;
        # Sandboxing
        ProtectSystem = "strict";
        ProtectHome = true;
        PrivateTmp = true;
        PrivateDevices = true;
        ProtectHostname = true;
        ProtectKernelTunables = true;
        ProtectKernelModules = true;
        ProtectControlGroups = true;
        LockPersonality = true;
        MemoryDenyWriteExecute = true;
        RestrictRealtime = true;
        PrivateMounts = true;

@Izorkin
Copy link
Contributor

Izorkin commented Apr 13, 2020

Found startup error:

апр 13 11:01:02 ___ systemd[1]: httpd.service: Main process exited, code=dumped, status=11/SEGV
апр 13 11:01:02 ___ systemd[1]: httpd.service: Failed with result 'core-dump'.
апр 13 11:01:02 ___ systemd[1]: httpd.service: Consumed 217ms CPU time, received 1.1K IP traffic, sent 8.7K IP traffic.
апр 13 11:01:02 ___ systemd-coredump[1043]: Process 676 (httpd) of user 54 dumped core.
апр 13 11:01:07 ___ systemd[1]: httpd.service: Scheduled restart job, restart counter is at 1.
апр 13 11:01:07 ___ systemd[1]: Stopped Apache HTTPD.
апр 13 11:01:07 ___ systemd[1]: httpd.service: Consumed 217ms CPU time, received 1.2K IP traffic, sent 8.7K IP traffic.
апр 13 11:01:07 ___ systemd[1]: Starting Apache HTTPD...
апр 13 11:01:07 ___ systemd[1]: httpd.service: Can't open PID file /run/httpd/httpd.pid (yet?) after start: No such file or directory
апр 13 11:01:07 ___ systemd[1]: Started Apache HTTPD.

[   10.214917] httpd[676]: segfault at 18 ip 00007fb17c516920 sp 00007ffc0b083ac8 error 4 in libpthread-2.30.so[7fb17c510000+f000]

@aanderse
Copy link
Member Author

ProtectHome = true;

Breaks enableUserDir.

At least a few of those systemd options will break many web applications I have to host so personally I won't be opening a PR to do that yet.

@aanderse
Copy link
Member Author

@Izorkin can you provide details from your configuration file please?

@Izorkin
Copy link
Contributor

Izorkin commented Apr 13, 2020

At least a few of those systemd options will break many web applications I have to host so personally I won't be opening a PR to do that yet.

Change to ProtectHome = false; or

+      sandboxWriteDir = mkOption {
+        type = types.listOf types.str;
+        default = [ "/var/www" "/var/data" ];
+        example = [ "/home/user/apache-www" ];
+        description = "
+          Allow writing files In directories to in sandboxing mode.
+        ";
+      };

...
         serviceConfig = {
...
+          # Access write directoryes
+          ReadWritePaths = [ "sandboxWriteDir" ];

@Izorkin can you provide details from your configuration file please?

  services.httpd = {
    enable = true;
    package = pkgs.apacheHttpd;
    adminAddr = "name@doname.com";
    enablePHP = true;
    mpm = "prefork";
    extraModules = [
      { name = "deflate";    path = "${pkgs.apacheHttpd}/modules/mod_deflate.so"; }
      { name = "status";     path = "${pkgs.apacheHttpd}/modules/mod_status.so"; }
      { name = "proxy_fcgi"; path = "${pkgs.apacheHttpd}/modules/mod_proxy_fcgi.so"; }
      { name = "remoteip";   path = "${pkgs.apacheHttpd}/modules/mod_remoteip.so"; }
    ];
    logFormat = "combined";
    extraConfig = ''
      AddDefaultCharset utf-8
      RemoteIPHeader X-Forwarded-For
        <IfModule mod_include.c>
          Options +Includes
          AddType text/html .shtml
          AddOutputFilter INCLUDES .shtm .shtml
        </IfModule>
      <Location /server-status>
        SetHandler server-status
        Order Deny,Allow
        Deny from all
        Allow from 127.0.0.1
      </Location>
    '';
    virtualHosts = {
      "_default_" = {
        enableSSL = false;
        documentRoot = "${pkgs.apacheHttpd}/htdocs";
        extraConfig = ''
          DirectoryIndex index.html
          <Directory "${pkgs.apacheHttpd}/htdocs">
            Order Deny,Allow
            Deny from all
            Allow from 127.0.0.1
          </Directory>
        '';
      };
      "_my_site" = {
        serverAliases = [ "www._my_site" ];
        forceSSL = true;
        enableACME = true;
        documentRoot = "/var/data/www";
        logFormat = "combined";
        extraConfig = ''
          DirectoryIndex index.php
          <Directory "/var/data/www">
              AllowOverride All
              <IfModule mod_deflate.c>
                SetOutputFilter DEFLATE
              </IfModule>
              <IfModule mod_include.c>
                Options +Includes
              </IfModule>
          </Directory>
        '';
      };
    };
  };

@aanderse
Copy link
Member Author

@Izorkin thank you for the config. I put together a configuration that uses a self signed certificate instead of enableACME and I'm unable to reproduce the error you have. Is there any more details you can provide, or maybe a nixos container to reproduce this?

@immae
Copy link
Contributor

immae commented Apr 14, 2020

Thanks for the work @aanderse!

Did you test your PR on your servers? I don’t see anything wrong here, but didn’t test it yet.

What comes from the top of my head:

  • privileged ports, but it is handled by the capability (I didn’t know it was possible !)
  • All my certificates are owned by root, so apache won’t be able to read anything. This is similar to nginx so I guess it is a good thing to push for anyway.
  • CGI’s are not limited to perl/php and may break as sometimes they run as other users, but I don’t have any such to test. I suppose the sane recommendations is now to have independent workers running and use sockets to talk with them, which is already handled.

I agree that any "sandboxing" option from systemd could have big consequences as far as httpd is concerned, since it needs access to directories which don’t belong to him or are at many places in the system. That would force to have a big systemd configuration to list them all (not infeasible though).

@Izorkin
Copy link
Contributor

Izorkin commented Apr 14, 2020

I am used custom phpPackage

  nixpkgs.overlays = [
    (self: super: {
      php = pkgs.nur.repos.izorkin.php71;
      phpPackages = pkgs.nur.repos.izorkin.php71Packages;
    })
  ];

With default phpPakage error:

[Tue Apr 14 10:53:10.042195 2020] [php7:error] [pid 731] [client 1.1.1.1:53627] PHP Fatal error:  Uncaught Error: Call to undefined function mysql_connect() in /var/data/www/classes/db/MySQL.php:51\nStack trace:\n#0 /var/data/www/classes/db/Db.php(319): MySQLCore->connect()\n#1 /var/data/www/classes/db/Db.php(239): DbCore->__construct()\n#2 /var/data/www/config/alias.php(66): DbCore::getInstance()\n#3 /var/data/www/classes/shop/Shop.php(329): pSQL()\n#4 /var/data/www/config/config.inc.php(114): ShopCore::initialize()\n#5 /var/data/www/index.php(27): require('/var/data/www/c...')\n#6 {main}\n  thrown in /var/data/www/classes/db/MySQL.php on line 51
[Tue Apr 14 11:22:46.065935 2020] [php7:error] [pid 2386] [client 1.1.1.1:63205] PHP Fatal error:  Uncaught Error: Call to undefined function mysql_connect() in /var/data/www/classes/db/MySQL.php:51\nStack trace:\n#0 /var/data/www/classes/db/Db.php(319): MySQLCore->connect()\n#1 /var/data/www/classes/db/Db.php(239): DbCore->__construct('2.2.2.2', 'askhost_sfair', 'U29xDQv6QbsH', 'askhost_sfair')\n#2 /var/data/www/config/alias.php(66): DbCore::getInstance()\n#3 /var/data/www/classes/shop/Shop.php(329): pSQL('sfair.ru')\n#4 /var/data/www/config/config.inc.php(114): ShopCore::initialize()\n#5 /var/data/www/index.php(27): require('/var/data/www/c...')\n#6 {main}\n  thrown in /var/data/www/classes/db/MySQL.php on line 51

Not working php mysqliSupport and mysqlndSupport

This code not worked:

    phpPackage =
        let
      base = pkgs.php74;
    in
      base.buildEnv {
        extensions = e: with e;
          base.enabledExtensions ++ [
            apcu redis memcached imagick
          ];
      };

Error:

building all machine configurations...
error: attribute 'override' missing, at /home/user/works/src-nix/nixpkgs/nixos/modules/services/web-servers/apache-httpd/default.nix:15:9
(use '--show-trace' to show detailed location information)
Traceback (most recent call last):
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/bin/..nixops-wrapped-wrapped", line 991, in <module>
    args.op()
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/bin/..nixops-wrapped-wrapped", line 412, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate)
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1052, in run_with_notify
    f()
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1003, in _deploy
    self.configs_path = self.build_configs(dry_run=dry_run, repair=repair, include=include, exclude=exclude)
  File "/nix/store/c13kpphf02i8habpc985axw3m0n0x815-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 671, in build_configs
    raise Exception("unable to build all machine configurations")
Exception: unable to build all machine configurations
Time: 0h:00m:03s

@aanderse
Copy link
Member Author

@immae great feedback. To address your points:

  • I haven't thoroughly tested this yet, so it remains indraft mode. Even once I have tested this thoroughly I will need other people to test this as well. Fortunately it is very easy to teston NixOS 20.03. I'll post instructions in the near future.
  • I'm glad you could learn something
  • Maybe you run some really sketchy old perl code using mod_perl as the wwwrun user and have a wildcard ssl certificate. You may or may not want to run as an unprivileged user in really exceptional and crazy situations 🤷‍♂️ The release notes mention this, and I made it exceptionally easy to continue to run the master process as root. Switching back and forth any number of times is supported too.
  • I haven't done cgi testing yet but it is on my list. Being on my list isn't enough, though. httpd is so massive and so flexible two absolute requirements on my mind are: a good number of people other than myself need to test this, and we need to support running the master process as root because this change doesn't work with every possible scenario I can conceive. It is a good and sane default, though.
  • I would probably consider merging a PR that added some sandboxing so long as it was well documented how to undo it for users. A bunch of code I maintain wouldn't work, so personally I have zero motivation at the the moment.

@aanderse
Copy link
Member Author

@Izorkin this will work very soon, waiting for #85026 to be merged and then will work.

I'm not familiar with adding NUR can you give me easy way to set phpPackage so I can reproduce?

Thank you so much for testing @Izorkin! Your feedback is always so valuable!

@Izorkin
Copy link
Contributor

Izorkin commented Apr 14, 2020

I'm not familiar with adding NUR can you give me easy way to set phpPackage so I can reproduce?

Need add:

nixpkgs.config.packageOverrides = pkgs: rec {
  nur = import (builtins.fetchTarball "https://github.com/nix-community/NUR/archive/master.tar.gz") { inherit pkgs; };
];

@immae
Copy link
Contributor

immae commented Apr 14, 2020

* I haven't done `cgi` testing yet but it is on my list. Being on my list isn't enough, though. `httpd` is so massive and so flexible two absolute requirements on my mind are: a good number of people other than myself need to test this, and we need to support running the master process as `root` because this change doesn't work with every possible scenario I can conceive. It **is** a good and sane default, though.

@aanderse
Okay I missed some important information in my first message: I thought that httpd was running as root only during startup, but it turns out that the "master" thread is always root, only its child are unpriviledged. This explains many things, especially the fact that httpd is flexible enough to let a CGI script run as a different user than wwwrun.

Considering that, I would suggest to have an additional configuration parameter keepRoot and a slight addition in systemd configuration:

User = if cfg.keepRoot then "root" else cfg.user;

This way, it’s easy for someone who needs it (in the situation given above) to keep his setup in a working state after your change, and still allow people who don’t need privileged process at all to be safe in this side.

Does that seem like a good thing to you?

@aanderse
Copy link
Member Author

@immae I've now tried out a few different common cgi setups and they are running fine without a root main process. I'm not convinced a keepRoot option is a good idea as it really encourages users to develop web applications and software in a way that even upstream has declared as "legacy". I support a large amount of "legacy" software, so I want it to be easy to run the master process as root... but I don't want to feel like NixOS (or upstream) is encouraging me to do so. Having a single line of configuration required to run as root, systemd.services.httpd.serviceConfig.User = lib.mkForce "root";, seems like the perfect mix of "this isn't cumbersome" and "are you sure you want to do this?".

I called this branch httpd-2020 because it is the year 2020 and I want to promote server configuration that reflects this, not server configuration that would make you think it is 10-15 years ago 🤷‍♂️

@immae
Copy link
Contributor

immae commented Apr 15, 2020

@immae I've now tried out a few different common cgi setups and they are running fine without a root main process. I'm not convinced a keepRoot option is a good idea as it really encourages users to develop web applications and software in a way that even upstream has declared as "legacy".

Ah, if upstream considers it legacy then my suggestion is not relevant actually :)

@aanderse
Copy link
Member Author

@Izorkin in your example what branch are you running off of? When you say pkgs.apacheHttpd does pkgs refer to master, release-20.03, release-19.09, or something else?

@aanderse aanderse requested a review from jtojnar April 16, 2020 23:54
nixos/doc/manual/release-notes/rl-2009.xml Outdated Show resolved Hide resolved
nixos/doc/manual/release-notes/rl-2009.xml Outdated Show resolved Hide resolved
nixos/doc/manual/release-notes/rl-2009.xml Outdated Show resolved Hide resolved
@Izorkin
Copy link
Contributor

Izorkin commented Apr 17, 2020

@Izorkin in your example what branch are you running off of? When you say pkgs.apacheHttpd does pkgs refer to master, release-20.03, release-19.09, or something else?

Used master branch

@aanderse
Copy link
Member Author

@Izorkin I'm unable to reproduce this. Can you specify a minimal configuration.nix and nixpkgs checkout so I can use nixos-rebuild build-vm ... to exactly reproduce your configuration and error?

@aanderse aanderse marked this pull request as ready for review April 21, 2020 00:10
@aanderse
Copy link
Member Author

Originally I was planning on leaving this unmerged for some time to allow for testing, but upon further contemplation I am thinking to merge this in the next couple days. The major change is as simple as systemd.services.httpd.serviceConfig.User = lib.mkForce "root"; to undo and being in master will force more testing than might occur otherwise. We're early enough in the 20.09 release cycle that I feel confident this will be thoroughly tested in time.

If anyone has any major objections please speak up or in a few days time I'll be merging.

@Izorkin I'm still looking to reproduce your error (which is NUR related, presumably) so please get back to me when you can.

@Izorkin
Copy link
Contributor

Izorkin commented Apr 22, 2020

@aanderse in latest master branch this error not reproduced.

Group = cfg.group;
Type = "forking";
PIDFile = "${runtimeDir}/httpd.pid";
Restart = "always";
RestartSec = "5s";
RuntimeDirectory = "httpd httpd/runtime";
RuntimeDirectoryMode = "0750";
AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With only

AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];

capabilities:

CapInh: 0000000000000400
CapPrm: 0000000000000400
CapEff: 0000000000000400
CapBnd: 0000003fffffffff
CapAmb: 0000000000000400

With

AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
CapInh: 0000000000000400
CapPrm: 0000000000000400
CapEff: 0000000000000400
CapBnd: 0000000000000400
CapAmb: 0000000000000400

May add CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ]; ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the service runs as wwwrun then they should be no need for CapabilityBoundingSet. If the service runs a root (using wwwrun for children processes) it seems likely the sysadmin likely has needs we can't anticipate and using CapabilityBoundingSet might be overstepping. I really want to make it easy for a sysadmin to keep the current (and distro standard) behavior for httpd if they need to.

@flokli any thoughts on that?

@aanderse
Copy link
Member Author

@GrahamcOfBorg test upnp wordpress mediawiki

@aanderse aanderse merged commit 16ab837 into NixOS:master Apr 26, 2020
@aanderse aanderse deleted the httpd-2020 branch April 26, 2020 00:04
@edolstra
Copy link
Member

I noticed that Apache now starts as the wwwrun user, which means that SSL private keys now need to be readable by wwwrun. Isn't this less secure than the previous situation, where they only had to be readable by root?

@aanderse
Copy link
Member Author

That depends on a large number of factors*. If you have an important wildcard then you want to seriously consider the implications and weigh the risk. If you are using a certificate for a single domain, especially if it is provisioned by Let's Encrypt, then maybe you're much better off with the default module.

* Are you running code as the wwwrun user via mod_perl or mod_php vs running code as a separate dedicated user with cgi/cgid or php-fpm? If you are running code (vs a static site), what sort of code are you running and what are the potential exploits? Are you running httpd in a container of some sort, or on a server that runs a ton of other services, etc...

Ultimately you need to balance the number of privilege escalation vulnerabilities that httpd has had and you would expect to have in the future and ask yourself what can cause more damage: a privilege escalation vulnerability when your httpd server runs as root, or a certificate having some exposure as the wwwrun user when scripts are being run as the wwwrun user.

To provide an example for both sides consider:

  • if you're running httpd in a container with mod_php you should probably care less about privilege escalation vulnerabilities than sketchy scripts having full access to your ssl wildcard certificate -> you should consider running the master process as root
  • if you're running httpd on a machine with 15 sites all developed internally, each with their own ssl certificate provisioned by Let's Encrypt, and each site connecting to a separate php-fpm instance (with a separate user to go along with that) you have much lower risk of certificates -> you should consider running the master process as wwwrun

I was sure to make it very easy to switch between running the parent process as root or wwwrun because I know how important it can be to continue running a site as root. I maintain a reasonable number of servers running httpd and a significant number of them still run with the httpd parent process as root.

I hope this sufficiently addresses your concerns. Regardless, I'm happy to further discuss the implications and any concerns you continue to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants