nixos/syncoid: split in multiple systemd services and harden them #98455

ju1m · 2020-09-22T09:41:38Z

Motivation for this change

Currently all syncoid commands are listed into a single script inside a single systemd service, hence if one command fails all the following commands are not run.
Ping @lopsided98

Things done

Put each command into a dedicated systemd service.
Add interval option per command, defaulting to the global interval.
Hardening of the service:

$ systemd-analyze security syncoid-losurdo-home-julm-work.service
  NAME                                                        DESCRIPTION                                                                    EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                                            0.5
✓ User=/DynamicUser=                                          Service runs under a static non-root user identity                                     
✓ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service cannot change UID/GID identities/capabilities                                  
✓ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has no administrator privileges                                                
✓ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has no ptrace() debugging abilities                                            
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                               0.3
✓ RestrictNamespaces=~CLONE_NEWUSER                           Service cannot create user namespaces                                                  
✓ RestrictAddressFamilies=~…                                  Service cannot allocate exotic sockets                                                 
✓ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service cannot change file ownership/access mode/capabilities                          
✓ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service cannot override UNIX file/IPC permission checks                                
✓ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has no network configuration privileges                                        
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules                                                     
✓ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has no raw I/O access                                                          
✓ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes cannot change the system clock                                       
✗ DeviceAllow=                                                Service has a device ACL with some special devices                                  0.1
✗ IPAddressDeny=                                              Service does not define an IP address allow list                                    0.2
✓ KeyringMode=                                                Service doesn't share key material with other services                                 
✓ NoNewPrivileges=                                            Service processes cannot acquire new privileges                                        
✓ NotifyAccess=                                               Service child processes cannot alter service state                                     
✓ PrivateDevices=                                             Service has no access to hardware devices                                              
✓ PrivateMounts=                                              Service cannot install system mounts                                                   
✓ PrivateTmp=                                                 Service has no access to other software's temporary files                              
✓ PrivateUsers=                                               Service does not have access to other users                                            
✓ ProtectClock=                                               Service cannot write to the hardware clock or system clock                             
✓ ProtectControlGroups=                                       Service cannot modify the control group file system                                    
✓ ProtectHome=                                                Service has no access to home directories                                              
✓ ProtectKernelLogs=                                          Service cannot read from or write to the kernel log ring buffer                        
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules                                             
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)                                    
✓ ProtectSystem=                                              Service has strict read-only access to the OS file hierarchy                           
✓ RestrictAddressFamilies=~AF_PACKET                          Service cannot allocate packet sockets                                                 
✓ RestrictSUIDSGID=                                           SUID/SGID file creation by service is restricted                                       
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI                                  
✓ SystemCallFilter=~@clock                                    System call allow list defined for service, and @clock is not included                 
✓ SystemCallFilter=~@debug                                    System call allow list defined for service, and @debug is not included                 
✓ SystemCallFilter=~@module                                   System call allow list defined for service, and @module is not included                
✓ SystemCallFilter=~@mount                                    System call allow list defined for service, and @mount is not included                 
✓ SystemCallFilter=~@raw-io                                   System call allow list defined for service, and @raw-io is not included                
✓ SystemCallFilter=~@reboot                                   System call allow list defined for service, and @reboot is not included                
✓ SystemCallFilter=~@swap                                     System call allow list defined for service, and @swap is not included                  
✓ SystemCallFilter=~@privileged                               System call allow list defined for service, and @privileged is not included            
✓ SystemCallFilter=~@resources                                System call allow list defined for service, and @resources is not included             
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities                                  
✓ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has no audit subsystem access                                                  
✓ CapabilityBoundingSet=~CAP_KILL                             Service cannot send UNIX signals to arbitrary processes                                
✓ CapabilityBoundingSet=~CAP_MKNOD                            Service cannot create device nodes                                                     
✓ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has no elevated networking privileges                                          
✓ CapabilityBoundingSet=~CAP_SYSLOG                           Service has no access to kernel logging                                                
✓ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has no privileges to change resource use parameters                            
✓ RestrictNamespaces=~CLONE_NEWCGROUP                         Service cannot create cgroup namespaces                                                
✓ RestrictNamespaces=~CLONE_NEWIPC                            Service cannot create IPC namespaces                                                   
✓ RestrictNamespaces=~CLONE_NEWNET                            Service cannot create network namespaces                                               
✓ RestrictNamespaces=~CLONE_NEWNS                             Service cannot create file system namespaces                                           
✓ RestrictNamespaces=~CLONE_NEWPID                            Service cannot create process namespaces                                               
✓ RestrictRealtime=                                           Service realtime scheduling access is restricted                                       
✓ SystemCallFilter=~@cpu-emulation                            System call allow list defined for service, and @cpu-emulation is not included         
✓ SystemCallFilter=~@obsolete                                 System call allow list defined for service, and @obsolete is not included              
✓ RestrictAddressFamilies=~AF_NETLINK                         Service cannot allocate netlink sockets                                                
✓ RootDirectory=/RootImage=                                   Service has its own root directory/image                                               
✓ SupplementaryGroups=                                        Service has no supplementary groups                                                    
✓ CapabilityBoundingSet=~CAP_MAC_*                            Service cannot adjust SMACK MAC                                                        
✓ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service cannot issue reboot()                                                          
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree                      
✓ LockPersonality=                                            Service cannot change ABI personality                                                  
✓ MemoryDenyWriteExecute=                                     Service cannot create writable executable memory mappings                              
✓ RemoveIPC=                                                  Service user cannot leave SysV IPC objects around                                      
✓ RestrictNamespaces=~CLONE_NEWUTS                            Service cannot create hostname namespaces                                              
✓ UMask=                                                      Files created by service are accessible only by service's own user by default          
✓ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service cannot mark files immutable                                                    
✓ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service cannot lock memory into RAM                                                    
✓ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service cannot issue chroot()                                                          
✓ ProtectHostname=                                            Service cannot change system host/domainname                                           
✓ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service cannot establish wake locks                                                    
✓ CapabilityBoundingSet=~CAP_LEASE                            Service cannot create file leases                                                      
✓ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service cannot use acct()                                     
✓ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service cannot issue vhangup()                                                         
✓ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service cannot program timers that wake up the system                                  
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                                  0.1

→ Overall exposure level for syncoid-losurdo-home-julm-work.service: 1.0 OK 🙂

ju1m · 2020-10-27T12:09:22Z

This PR was broken by 733acfa . It is now rebased against current master with modifications to support the zfs allow calls from 733acfa

lopsided98

I think this is a good idea. I've tested this pretty thoroughly in my setup, and it works with the changes I suggested.

This is the modified version of this PR I have been testing: lopsided98@0894223

lopsided98 · 2020-11-10T02:16:05Z

nixos/modules/services/backup/syncoid.nix

+              ]) (localPoolName c.target);
+            serviceConfig.User = cfg.user;
+            serviceConfig.Group = cfg.group;
+            serviceConfig.Restart = mkDefault "on-failure";


I don't think this is a good default. With the default StartLimit settings, persistent failures cause the service to be restarted in a loop forever and never marked as failed. This prevents me from getting a notification that my backups are failing. If users want restarting they can easily configure it themselves with StartLimit settings appropriate for their use case.

Ok, I've removed Restart= and RestartSec=, but I do not have enough monitoring perspective to know if this is suitable or not like that.

nixos/modules/services/backup/syncoid.nix

lopsided98 · 2020-11-11T05:52:29Z

nixos/modules/services/backup/syncoid.nix

+            # For syncoid to be able to create /var/lib/syncoid/.ssh/
+            # and to use custom ssh_config or known_hosts.
+            home = "/var/lib/syncoid";
+            createHome = true;


I'd prefer StateDirectory = "syncoid" in the systemd service. I think it is the more idiomatic way of doing it.

For my use case, I actually preferred not having a stateful known_hosts file because it makes it harder to change host keys (I normally manage them declaratively but now I would also have to manually edit /var/lib/syncoid/.ssh/known_hosts as well). I'm fine with this change though if it helps your use case.

Sure, I've moved to StateDirectory= which ensures correct permissions and automatically binds the directory in RootDirectory=.

Concerning the use case of /var/lib/syncoid/.ssh/, when generating a /etc/ssh/ssh_known_hosts with:

programs.ssh.knownHosts = { mermet = { hostNames = [ "mermet" "mermet.sourcephile.fr" ]; publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFvKN2sIpH782MFjaOpcty1Hs/T/TPNJpXI08H3O3oxl"; }; };

and using this hostname in :

services.syncoid.commands."backup@mermet.sourcephile.fr:rpool/var/mail".target = "losurdo/backup/mermet/var/mail";

syncoid now writes a /var/lib/syncoid/.ssh/known_hosts specifying the IP address of the hostname:

80.67.180.129 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFvKN2sIpH782MFjaOpcty1Hs/T/TPNJpXI08H3O3oxl

instead of issuing the following warnings:

losurdo syncoid[19440]: Could not create directory '/var/empty/.ssh'. losurdo syncoid[19440]: Failed to add the ED25519 host key for IP address '80.67.180.129' to the list of known hosts (/var/empty/.ssh/known_hosts).

Hence, if that IP address is dynamic, having such a stateful known_hosts may be useful, though there are probably some --sshoption that can help.

lopsided98 · 2020-11-15T22:36:08Z

nixos/modules/services/backup/syncoid.nix

+                "+/run/booted-system/sw/bin/zfs" "allow"
+                cfg.user "create,mount,receive,rollback" pool
+              ]) (localPoolName c.target);
+            serviceConfig.User = cfg.user;


We need to set PrivateTmp = true, otherwise different commands that have the same destination host and run in parallel end up using the same SSH control socket. When one command finishes, it kills the connection of any others using the same socket.

Indeed, see also jimsalterjrs/sanoid#532 (comment).
I've set PrivateTmp=true and also taken the time to find working values for the security options, based upon systemd-analyze security syncoid-$NAME.service and previous experiences like #98904 .
As always, it might still be too much hardening. Please bear with me.

The hardening doesn't cause any problems for me, although as always it is definitely an increased maintenance burden that requires in depth systemd knowledge to understand. I'd like to get someone else's opinion, although it seems to be nearly impossible to find reviewers for sanoid/syncoid PRs for some reason.

I'm trying out sanoid/syncoid, hopefully I'll have the experience in a few weeks to have actual opinions. I'll try to remember this (and other PRs) -- but feel free to ping me if it's still stuck in a while.

Thanks, it would be great to get this merged. Make sure you use #83904, otherwise sanoid templates won't work right.

lopsided98

The NixOS test needs to be updated to account for the changed syncoid service name.

nixos-discourse · 2020-11-25T06:49:03Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/385

ju1m · 2021-05-10T15:06:08Z

Hardened with ProcSubset=pid and ProtectProc=invisible as I've done elsewhere.

ju1m · 2021-06-15T21:31:40Z

Rebased against latest master, solving merge conflit with d87903a

lopsided98

@lheckemann I've been running the latest version of this PR since last night without problems, so I'm good with merging this now if you want, as we discussed on IRC.

etu · 2021-07-24T06:45:24Z

@ju1m

I've tested this, it seems to work, but some of the locking down seems to cause warnings (not errors though).

Jul 24 08:42:26 fenchurch zed[30343]: eid=179 class=history_event pool='zroot'
Jul 24 08:42:27 fenchurch syncoid[30341]: WARN: lzop not available on source ssh:-S /tmp/syncoid-root@vps05.elis.nu-1627108946 root@vps05.elis.nu- sync will continue without compression.
Jul 24 08:42:27 fenchurch syncoid[30341]: WARN: mbuffer not available on source ssh:-S /tmp/syncoid-root@vps05.elis.nu-1627108946 root@vps05.elis.nu - sync will continue without source buffering.
Jul 24 08:42:28 fenchurch syncoid[30383]: Error: /proc must be mounted
Jul 24 08:42:28 fenchurch syncoid[30383]:   To mount /proc at boot you need an /etc/fstab line like:
Jul 24 08:42:28 fenchurch syncoid[30383]:       proc   /proc   proc    defaults
Jul 24 08:42:28 fenchurch syncoid[30383]:   In the meantime, run "mount proc /proc -t proc"
Jul 24 08:42:28 fenchurch syncoid[30341]: NEWEST SNAPSHOT: autosnap_2021-07-24_06:30:07_frequently
Jul 24 08:42:28 fenchurch syncoid[30393]: Error: /proc must be mounted
Jul 24 08:42:28 fenchurch syncoid[30393]:   To mount /proc at boot you need an /etc/fstab line like:
Jul 24 08:42:28 fenchurch syncoid[30393]:       proc   /proc   proc    defaults
Jul 24 08:42:28 fenchurch syncoid[30393]:   In the meantime, run "mount proc /proc -t proc"

This is not the case when pushing the snapshots to a remote host, only when pulling snapshots from a remote host.

ju1m · 2021-07-24T09:20:34Z

@etu, oh, right, this is an important warning: syncoid calls ps to detect whether there is already a zfs receive on the receiving dataset. This is caused by my recent addition of ProcSubset=pid:

$ sudo systemd-run --pipe -p DynamicUser=true -p ProcSubset=pid ps -Ao args=
Running as unit: run-u10883.service
Error: /proc must be mounted
  To mount /proc at boot you need an /etc/fstab line like:
      proc   /proc   proc    defaults
  In the meantime, run "mount proc /proc -t proc"

I am also relaxing ProtectProc=invisible back to ProtectProc=default to let ps have access to all the processes.
Thanks!

phryneas · 2022-02-19T14:52:56Z

Could it be that this has hardened the service so hard it has no more access to the ssh key in the end? I moved the ssh key into /etc in the end, which seems to work, but everywhere else it just fails.

ju1m · 2022-02-19T19:36:23Z

@phryneas yes, this PR chrooted syncoid in a temporary RootDirectory=, hence if you do not put your ssh key in one of the following directories (or one of their sub-directories): [ builtins.storeDir "/etc" "/run" ]; the key will not be accessible to syncoid. If you need to use a custom directory, you can add it to config.services.syncoid.service.serviceConfig.BindReadOnlyPaths.

Note that #147559 (awaiting reviews) uses the new systemd option LoadCredential= which takes care to mount the key in the RootDirectory=.

phryneas · 2022-02-19T20:53:24Z

Thank you @ju1m, config.services.syncoid.service.serviceConfig.BindReadOnlyPaths = [ "/var/lib/remote_backup/.ssh" ]; was the missing puzzle piece for me to make things work - and I learned something new about systemd (and how I just can add extra things to systemd services - I didn't know I could just add stuff to config.services.syncoid.service.serviceConfig).

I'm going to subscribe that future PR.

ju1m · 2022-02-20T08:14:39Z

@phryneas , note that you would usually reach serviceConfig through something like config.systemd.services.${serviceName}.serviceConfig but because in the case of syncoid.service many systemd services are generated this convenient option enables to easily set common settings for all those systemd services.

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Sep 22, 2020

ju1m force-pushed the syncoid-split branch from 9fabcf4 to 517c7e7 Compare September 22, 2020 15:31

ju1m mentioned this pull request Sep 23, 2020

[syncoid] CRITICAL ERROR: snapshots couldn't be listed jimsalterjrs/sanoid#532

Closed

ju1m force-pushed the syncoid-split branch from 517c7e7 to e460a70 Compare October 27, 2020 03:34

ju1m force-pushed the syncoid-split branch from e460a70 to f343f13 Compare October 28, 2020 09:39

ju1m force-pushed the syncoid-split branch from f343f13 to 1f26e51 Compare November 6, 2020 20:20

lopsided98 suggested changes Nov 22, 2020

View reviewed changes

ju1m force-pushed the syncoid-split branch from 1f26e51 to 1f94f2f Compare November 25, 2020 05:36

ju1m changed the title ~~nixos/syncoid: split in multiple systemd services~~ nixos/syncoid: split in multiple systemd services and harden them Nov 25, 2020

ju1m force-pushed the syncoid-split branch from 1f94f2f to a19613e Compare May 10, 2021 15:04

lopsided98 mentioned this pull request May 20, 2021

sanoid: 2.0.3 -> 2.1.0 #123644

Merged

10 tasks

ju1m force-pushed the syncoid-split branch from a19613e to ab2ff7e Compare May 20, 2021 06:04

ofborg bot removed the 10.rebuild-linux: 1 label May 20, 2021

SuperSandro2000 requested a review from aanderse May 22, 2021 15:58

ju1m force-pushed the syncoid-split branch from ab2ff7e to 250731a Compare June 15, 2021 21:29

ju1m force-pushed the syncoid-split branch from 250731a to f81f7e3 Compare June 16, 2021 00:08

ju1m requested review from lopsided98 and srhb June 27, 2021 00:41

ju1m requested a review from etu July 24, 2021 01:50

lopsided98 approved these changes Jul 24, 2021

View reviewed changes

nixos/syncoid: split in multiple systemd services

d05a1ab

ju1m force-pushed the syncoid-split branch from f81f7e3 to d05a1ab Compare July 24, 2021 09:26

etu approved these changes Jul 24, 2021

View reviewed changes

etu merged commit 6984e68 into NixOS:master Jul 24, 2021

etu mentioned this pull request Aug 9, 2021

nixos/{syncoid,sanoid}: Improve ZFS permission delegation #131118

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/syncoid: split in multiple systemd services and harden them #98455

nixos/syncoid: split in multiple systemd services and harden them #98455

ju1m commented Sep 22, 2020 •

edited

ju1m commented Oct 27, 2020

lopsided98 left a comment

lopsided98 Nov 10, 2020

ju1m Nov 25, 2020

lopsided98 Nov 11, 2020

ju1m Nov 25, 2020 •

edited

lopsided98 Nov 15, 2020

ju1m Nov 25, 2020

lopsided98 Dec 5, 2020

srhb Feb 27, 2021

lopsided98 Feb 27, 2021

lopsided98 left a comment

nixos-discourse commented Nov 25, 2020

ju1m commented May 10, 2021

ju1m commented Jun 15, 2021

lopsided98 left a comment

etu commented Jul 24, 2021

ju1m commented Jul 24, 2021

phryneas commented Feb 19, 2022

ju1m commented Feb 19, 2022

phryneas commented Feb 19, 2022

ju1m commented Feb 20, 2022

nixos/syncoid: split in multiple systemd services and harden them #98455

nixos/syncoid: split in multiple systemd services and harden them #98455

Conversation

ju1m commented Sep 22, 2020 • edited

Motivation for this change

Things done

ju1m commented Oct 27, 2020

lopsided98 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ju1m Nov 25, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lopsided98 left a comment

Choose a reason for hiding this comment

nixos-discourse commented Nov 25, 2020

ju1m commented May 10, 2021

ju1m commented Jun 15, 2021

lopsided98 left a comment

Choose a reason for hiding this comment

etu commented Jul 24, 2021

ju1m commented Jul 24, 2021

phryneas commented Feb 19, 2022

ju1m commented Feb 19, 2022

phryneas commented Feb 19, 2022

ju1m commented Feb 20, 2022

ju1m commented Sep 22, 2020 •

edited

ju1m Nov 25, 2020 •

edited