[RFC 0059]: Systemd Service Secrets #59

d-goldin · 2019-11-16T16:14:22Z

Hi,

This is a first draft for an RFC about a possible approach to secrets management for systemd services.

Curious to hear your thoughts.

Rendered

rfcs/0058-secrets-for-services.md

turion

I find this a great idea in total, and one which would bring NixOS a step closer to a pain-free devops' choice.

Side note: This approach does not prevent one service from reading the secrets of another, e.g. through privilege escalation or remote code execution. But I think that issue is beyond the scope of this RFC, and should better be handled with something like containers.

rfcs/0058-secrets-for-services.md

turion · 2019-11-17T11:45:57Z

rfcs/0058-secrets-for-services.md

+# Drawbacks
+[drawbacks]: #drawbacks
+
+I can't really think of a serious drawback right now, but hopefully the


How does the secret store deal with system/main config upgrades? When I uninstall an old service and install a new one that asks for a secret of the same name, does the new service get access to the old secret?
When I uninstall a lot of services, are old secrets garbage collected? Do downgrades always work seamlessly?

Can you outline the possible pitfalls regarding that topic in an extra section?

In the current shape, no such management would happen. The secrets would just remain in the secrets store and if a config author decides to re-use them, then so be it. I can include that. I was initially thinking of the drawbacks of "sth that would be worse when adding this", but I guess it depends on the reference point. Definitely good points to add, even if they would remain unhandled.

Well, it's maybe a strawman, but if you put a secret in /nix/store, it's going to vanish upon garbage collection when I uninstall the package. If I don't care whether it's readable by root or by all users (e.g. inside a container), this is a drawback now because no such garbage collection would take place.

As in "something that would be worse when adding this", you could also say that people possibly expect that NixOS doesn't do any state management by default and that downgrading will bring their machine back into the exact state, but that's not the case anymore.

you could also say that people possibly expect that NixOS doesn't do any state management by default

The difference being that before this RFC, I'd manually handle the secret state and remember to handle it. After all, I'm deleting the line saying service.foo.secretsFile = "/foo/bar";, and when I delete the line, I'll delete the file. This is now not the case anymore. Instead, I need to remember how this semiautomatic secret handling works.

if you put a secret in /nix/store

It is exactly the purpose of this RFC not to have secrets in the nix store. This is possible without any problems and if they are referenced in the nixos config they would not be garbage collected as currently is the case. And this can still be achieved with this proposal as before.

If I don't care whether it's readable by root or by all users (e.g. inside a container)

Actually currently the nix store of (nixos) containers is shared with the host and all other containers, so that might be an issue nonetheless.

With regard to secret state handling, a lot of services have the possibility do use a passwordFile, secretFile, etc. but there is no abstraction for handing out permissions to secrets correctly, which is something like the proposal in this RFC is needed. Currently you have to not only drop the files on the machine yourself in the correct destination (same with this RFC, as long as you are using a fetcher that requires that), but also set the correct permissions for the respective service, which might be a chicken-and-egg problem of adding users for service by enabling it and the service requiring the secret.

rfcs/0058-secrets-for-services.md

turion · 2019-11-17T11:50:27Z

rfcs/0058-secrets-for-services.md

+* When using a scope with multiple services, ideally only the secrets
+  referenced in the services definition should be made available to each
+  service. Right now all the secrets of the scope are blindly copied.
+* Transition of most critical services to use proposed approach


I think it would be better to transition some non-critical services first, to gain some experience without breaking anything. If this works for 10+ regularly used non-critical e.g. webapps, then transition everything to this approach.

I will fix the wording on that one. Parts of this section are maybe still a bit too "note to self"-like. What I intended was not for "inclusion into upstream" but more to hypothetically verify this is covering all the most important aspects needed for the most critical services. Your version is definitely better for an actual slow inclusion into upstream step.

Co-Authored-By: Manuel Bärenz <programming@manuelbaerenz.de>

Ericson2314 · 2019-11-18T00:21:41Z

The detailed design is quite long. I think it's better to clean up the Nix secrets story before moving downstream, and I suspect trying to do everything at once will leave us stuck.

globin · 2019-11-18T09:17:58Z

@Ericson2314 I think this proposal is orthogonal to the secrets-in-nix-store issue and that this is quite a nice way to move forward without getting in the way of that, as this RFC abstracts over where the secrets are fetched from. In the case of nix store secrets being implement one could easily add a fetcher to use that, but also keep the abstraction in place for remote secret management systems if for some reason you are required to use that.

But obviously I'd be very happy if #5 or similar would be continued!

globin · 2019-11-18T09:58:50Z

Side note: This approach does not prevent one service from reading the secrets of another, e.g. through privilege escalation or remote code execution. But I think that issue is beyond the scope of this RFC, and should better be handled with something like containers

@turion I'm not sure how you mean this? This as far as I can see is possible with this RFC as you can also use non-FS fetcher functions. And obviously always depends on how far you want to go with isolation.

turion · 2019-11-18T10:38:20Z

Side note:

It was really just a comment that all secrets are still root-readable, and that this RFC is orthogonal to more isolation/virtualization.

aanderse · 2019-11-18T11:20:50Z

From reading this over it seems once a module uses this proposal there is no opt out for the user, correct?

If say the drawback is that this adds more complexity to the system. When I was new to nix I always enjoyed being able to read a modules source and usually understand what is happening... translate things into what a regular distro does. This is an entirely new abstraction that has no equivalent in a regular distro and after reading through it twice I still feel very uncomfortable thinking about how I can explain/justify this mechanism to my co-workers who manage nixos servers with me, but have not dived into nixos enough to understand all the nuances yet. I foresee having to answer many questions to puzzled looks from people wondering why we can't continue to keep doing what we're doing, and wondering why, as non "nix people", they no longer understand module source code.

I also wonder if a product similar to vault should be put in the alternatives section. I'm under the impression that while this may not be an entirely solved problem, there are existing solutions out there that can fit.

globin · 2019-11-18T11:48:35Z

I also wonder if a product similar to vault should be put in the alternatives section. I'm under the impression that while this may not be an entirely solved problem, there are existing solutions out there that can fit.

I think that this rather allows us to consistently use something like vault or some otherwise created secrets folder etc. and not having to depend on every module implementing support for it. It does create a further abstraction but by that allows all modules to have a consistent way of passing secrets and not a different implementation for each of them.

aanderse · 2019-11-18T12:00:59Z

@globin I meant a single solution like vault could be chosen by the sysadmin for every secret on a machine and then module code could continue as is, unmodified. This provides greater flexibility, though currently at the cost of upfront work for the sysadmin. That being said I concede that the open issues regarding secrets in the nix store have been open for years without resolution, and no one has provided a nice to use module for something like vault that integrates nicely into nixos yet.

I guess some sort of system/module that keeps this logic out of the module system would have been my desired outcome.

I hope my opinion has provided at least minimal value as a differing point of view.

globin · 2019-11-18T12:31:32Z

The thing is that is not possibly currently.

For all modules that only allow passing secrets by string (secret content) you'd have to modify it to pass in the secret differently as it would otherwise be added to the nix store nonetheless.

This change should happen anyway even without this proposal and a lot of modules now at least allow passing by file, where one can choose to use a non-nix-store-file with more specific permissions or just a pkgs.writeText in one does not care. For the ability to use something like vault one would still have to create new services, similar to the proposal here. That's why I think it would be nice to have an abstraction so that we can standardise all services handling secrets to have a consistent interface.

aanderse · 2019-11-18T13:14:32Z

@globin I entirely agree that all modules need to replace any password options with passwordFile (or the like) regardless of this RFC, and that is something I (along with many others) have slowly been working toward. I also agree that pkgs.writeText in conjunction with passwordFile is the proper justification for entirely eliminating any password like options, like I proposed when I removed the dbPassword option from the zabbixServer module.

d-goldin · 2019-11-18T22:22:25Z

@aanderse:

From reading this over it seems once a module uses this proposal there is no opt out for the user, correct?

Do you mean for a module author or a user? If a module is switched to this, then no, without "more involved" fiddling about a user wouldn't be able to change that unless author has created multiple avenues. We could of-course add a "secret store" that is just the nix store as some sort of fallback for scenarios where people don't care and they can at least go and add their secrets to the nix store as before.

If say the drawback is that this adds more complexity to the system. [...]. I foresee having to answer many questions to puzzled looks from people wondering why we can't continue to keep doing what we're doing, and wondering why, as non "nix people", they no longer understand module source code.

This is of course different perceptions, but for me NixOS is already quite an opinionated distribution that does a lot of things quite differently and brings a few abstractions that no other distro has.
Imho pretty much anything new of this sort will have the issue of change/surprise/added complexity.

I wonder, why does this particular code/interface seem more complex than for instance all the module evaluation and merging logic in https://github.com/NixOS/nixpkgs/blob/master/lib/modules.nix? It is of course possible to simplify and document to help people understand it more easily, once we understand what points of confusion exist.

I also wonder if a product similar to vault should be put in the alternatives section. I'm under the impression that while this may not be an entirely solved problem, there are existing solutions out there that can fit.

Well, yes. I thought of this as a given, that people can still go and use vault directly if they want/their software supports it. If all things we are dealing with would support sth like vault, then it would be great. But as @globin said, we'd still want the ability to plug in another service to manage secrets if need be, so some abstraction would likely emerge.

I guess some sort of system/module that keeps this logic out of the module system would have been my desired outcome.

Could you please explain a little bit what exactly you mean? Something like this, but being just some library living separately from NixOS in general, or is this more about how such an approach would be integrated with the module system? For instance, instead of the modules using this interface directly, the user of a module calling some functions on it?

Thanks, I will keep the points in mind when I do the next pass.

rfcs/0058-secrets-for-services.md

Co-Authored-By: Lassulus <github@lassul.us>

rfcs/0058-secrets-for-services.md

edolstra · 2019-11-20T16:34:06Z

rfcs/0058-secrets-for-services.md

+          DynamicUser = true;
+        };
+      };
+    };


This seems unnecessarily verbose. I would do something like this:

systemd.services.foo = { needsSecrets = [ "secret1" ]; ... };

and then our systemd module can generate whatever helper units are necessary.

It also avoids imperative-sounding function names like mkSecretsScope which suggest that they allocate a unique new scope, but that's not the case, e.g. in

secretsScope1 = mkSecretsScope { loadSecrets = [ "secret1" "secret2" ]; type = "folder"; }; secretsScope2 = mkSecretsScope { loadSecrets = [ "secret1" "secret2" ]; type = "folder"; };

secretsScope1 and secretsScope2 are actually the same scope.

This seems unnecessarily verbose. I would do something like this: [...]

I agree, it's more compact. Why I initially decided against it and for something that modifies the resulting structure separately was to reduce changes to the existing systemd modules. At the same time I thought it would be nice to be able to get the secrets as arguments, which I thought made it nicer to deal with in code. In the needsSecrets case, if I understand it correctly, the user would be requesting a secret by its name, like in the scope creation, but then would have to possibly deal with paths (as strings) to pass the secret as an argument to the service, or load it into an env-var, because there would not be an automatic mapping mechanism anymore. Unless we pack it up into a shell variable or so.

It also avoids imperative-sounding function names like mkSecretsScope which suggest that they allocate a unique new scope, but that's not the case, e.g. ...

I do not necessarily perceive mk* as an imperative terminology, but that depends on the reader. We have a lot of pure mk* functions that construct some structure. But I'm not at all attached to the naming here, so we can change it to whatever seems more suitable if they're still around further down the road.

Edit: In fact, I'm not attached to most of the terms in the RFC, such as "sidecart" and similar. I merely picked them from what I thought would make it easily enough understood. So if there are suggestions to rename things, I'm up for it.

@edolstra: Did this sufficiently address your remarks? I'm not super familiar with the inner workings of the process, so for now I just left those as discussion comments here, but I'm willing to incorporate some of the things pointed out into "alternatives" or the core section, if there is some consensus around that.

So, getting back to this suggestion - if somebody could point me to the simplest way of implementing it with such an additional field on any systemd service without needing to change too many things in the guts of core modules I'm willing to try and adjust the poc to see how that works. One thing I do kinda like though is the ability to have some at least very basic checking of secrets used, which wouldn't work the same way with just strings.

edolstra · 2019-11-20T17:04:07Z

rfcs/0058-secrets-for-services.md

+  with secrets
+* "Side-car" service: A privileged systemd service running the fetcher
+  function to retrieve the secret, and initially create the service
+  namespace


It's not clear to me why the helper unit is needed. Can't the keys be fetched using an ExecStartPre=+... command?

More generally, instead of creating our own mechanism for passing keys to services, maybe the kernel keyring mechanism can be used for this? Units would call keyctl request/search/read/... to fetch keys. These keys would either be preloaded into the keyring or produced on demand using the request-key program.

I briefly tried ExecStartPre, but unfortunately it does not seem to run within the mount namespace, so it doesn't have a access to the PrivateTmp we want. I am not sure if this is intentional or not.

Regarding kernel keyring mechanism - I agree, it might be a good default backend. The directory based thing was just the dumbest proof-of-concept case I came up with (given that similar approaches are used in nixops and krops/stockholm). Part of the intention is to have a somewhat agnostic interface.

Kernel keyring mechanism sounds overkill to me. Its usecase is not to communicate files between userland processes, but between userland and kernel drivers. Files are a perfectly sufficient abstraction for passing secrets around in userland.

I have mixed feelings about this.

Using files usually implies having to worry about ACLs and who's allowed to access them. We can cheat by mounting them in a private namespace, but it's still a bit cumbersome.
Sometimes you want to have "use once" properties and provide a new key on every read / issue tokens on access etc. We could cheat again by providing these key files by a fuse filesystem, but then it just gets more complicated.

The kernel keyring might be a good abstraction over all this, it's just not widely adapted currently and lacking real-world usage. Reading keyrings (7) looks promising. In addition to thread/process/session, there's also an upcall feature, bouncing back to userspace to request secrets which could be a request to whatever credentials provider is used.

I'd love to experiment with that a bit, or see some real-world examples. Anybody aware of these?

kernel keyring is very much intended for userspace keys too. See the kerberos stuff that has been ported to use it for that, or systemd's cryptsetup.

I think the kernel keyring has deficiencies (upcalls, yuck! also no namespacing for containers, …), but it probably is the right approach in the long run.

I briefly tried ExecStartPre, but unfortunately it does not seem to run within the mount namespace, so it doesn't have a access to the PrivateTmp we want. I am not sure if this is intentional or not.

I think that's what the ! prefix is for:

"!" - Similar to the "+" character discussed above this permits invoking command lines with elevated privileges. However, unlike "+" the "!" character exclusively alters the effect of User=, Group= and SupplementaryGroups=, i.e. only the stanzas that affect user and group credentials. Note that this setting may be combined with DynamicUser=, in which case a dynamic user/group pair is allocated before the command is invoked, but credential changing is left to the executed process itself.

Lassulus · 2020-05-14T13:31:58Z

@dhess did you post a summary of the meetup somewhere else? I can't find one

dhess · 2020-05-14T13:38:26Z

@Lassulus Sorry, I haven't posted the meeting notes yet. I'll get to it soon.

dhess · 2020-05-16T12:31:48Z

Sorry for the delay in posting the meeting minute notes, but here they are.

Notes from 2020-05-06 call

Attending:

@d-goldin reported the following:

Nobody has stepped up to co-author yet.
There's now a proof-of-concept repo: https://github.com/d-goldin/nix-svc-secrets/blob/master/lib/secrets.nix
The Hashicorp Vault implementation in this proof-of-concept implementation isn't very robust at this point.
@poettering's proposal needs more review, but provides some good primitives. It probably doesn't solve all the NixOS problems, e.g., if we want Vault we'd still need some way of integrating with it. It only provides a few mechanisms. How does a user of a service module describe where secrets would live

@dhess & @flokli agreed to try out the implementation before the next meeting.

@flokli:

Will open a discussion with @poettering. NixOS should be involved from the start.
How we expose any systemd hooks from the NixOS module system still needs to be solved.
Will ask @poettering about other mechanisms, e.g., Vault.

Given @poettering's intention to work on related things in systemd, a vote was taken on this RFC, whether to proceed with the work, pause it for a bit until systemd work is better fleshed-out, or close the proposal. The votes were as follows (@d-goldin did not vote):

@dhess: continue
@aanderse: pause for a month or two
@Lassulus: continue
@flokli: continue
@andir: continue

The vote was 4 : 1 : 0 (continue : pause : close), so we'll proceed with the RFC in parallel with the systemd work. Any further work should be done with the intention of integrating with systemd's primitives (if any) when the time comes.

d-goldin · 2020-05-21T10:11:13Z

@dhess: s/@globin/@andir

dhess · 2020-05-21T11:22:32Z

@d-goldin Thank you. I've updated the notes. @andir I'm sorry!

dudebout

Noting a couple typos.

rfcs/0059-secrets-for-services.md

Mic92 · 2020-06-25T11:03:34Z

@dhess What are next steps for this RFC. Another meeting?

dhess · 2020-06-25T11:18:03Z

@Mic92 Yes, eventually, but not much has changed since the last one, so I don't know if it's warranted at this time. I haven't had any time to try out the proof-of-concept implementation yet.

@flokli How about you, have you tried it? Also, is there any update on the discussions with upstream?

aanderse · 2020-07-12T17:29:41Z

Thought I would link this here for future reference. Possibly of interest to people subscribed to this thread: https://cedwards.xyz/protecting-systemd-service-secrets/

Mic92 · 2020-07-23T14:18:21Z

We have now a pass based secret implementation and one based on sops.

d-goldin · 2020-08-02T13:06:13Z

@dhess, @flokli: Have you had a chance to give it a try, as a follow-up from the last call?

d-goldin · 2020-08-18T18:57:06Z

As things stand, I don't think this will progress much, so I propose to close this.

d-goldin · 2020-08-25T17:26:48Z

Thanks for everyone's feedback, ideas and other contributions. I hope they'll be useful at the very least as a reference for other efforts.

Mic92 · 2020-08-26T19:40:39Z

Systemd actually has this feature now: systemd/systemd#16568
It just got merged and we could start using it in NixOS as well.

d-goldin changed the title ~~rfc-0057: systemd service secrets~~ [RFC-0058]: Systemd Service Secrets Nov 16, 2019

rfc-0058: systemd service secrets

00c0dd3

d-goldin force-pushed the master branch from 538f402 to 00c0dd3 Compare November 16, 2019 18:18

globin mentioned this pull request Nov 16, 2019

Provide options for storing secrets outside the Nix store NixOS/nixpkgs#24288

Open

75 tasks

turion reviewed Nov 17, 2019

View reviewed changes

rfcs/0058-secrets-for-services.md Outdated Show resolved Hide resolved

turion reviewed Nov 17, 2019

View reviewed changes

rfcs/0058-secrets-for-services.md Outdated Show resolved Hide resolved

turion reviewed Nov 17, 2019

View reviewed changes

d-goldin and others added 4 commits November 17, 2019 23:28

Typo fix

0942ab5

Co-Authored-By: Manuel Bärenz <programming@manuelbaerenz.de>

formatting fix

8cd4e84

Co-Authored-By: Manuel Bärenz <programming@manuelbaerenz.de>

Typo fix

4b658f8

Co-Authored-By: Manuel Bärenz <programming@manuelbaerenz.de>

simplify function to retrieve fetcher function

8a0ed5b

Co-Authored-By: Manuel Bärenz <programming@manuelbaerenz.de>

Lassulus reviewed Nov 19, 2019

View reviewed changes

rfcs/0058-secrets-for-services.md Outdated Show resolved Hide resolved

typo fix

ea81bf7

Co-Authored-By: Lassulus <github@lassul.us>

edolstra added status: new need co-author labels Nov 20, 2019

edolstra reviewed Nov 20, 2019

View reviewed changes

rfcs/0058-secrets-for-services.md Outdated Show resolved Hide resolved

pointing to the right issue NixOS#8

9067c85

edolstra reviewed Nov 20, 2019

View reviewed changes

edolstra added status: open for nominations Open for shepherding team nominations and removed status: new labels Nov 21, 2019

lheckemann mentioned this pull request May 28, 2020

RFC meeting 2020-05-28 NixOS/rfc-steering-committee#37

Closed

10 tasks

edolstra mentioned this pull request Jun 11, 2020

RFC meeting 2020-06-11 NixOS/rfc-steering-committee#38

Closed

9 tasks

dudebout suggested changes Jun 13, 2020

View reviewed changes

rfcs/0059-secrets-for-services.md Outdated Show resolved Hide resolved

rfcs/0059-secrets-for-services.md Outdated Show resolved Hide resolved

aanderse mentioned this pull request Jun 16, 2020

nixos/mysql: enable sandbox mode NixOS/nixpkgs#87833

Merged

10 tasks

Fixing a few typos

6ded518

Mic92 mentioned this pull request Jun 25, 2020

RFC meeting 2020-06-25 NixOS/rfc-steering-committee#39

Closed

12 tasks

edolstra changed the title ~~[RFC-0059]: Systemd Service Secrets~~ [RFC 0059]: Systemd Service Secrets Jul 9, 2020

CMCDragonkai mentioned this pull request Jul 13, 2020

Exposing different interfaces for use of secrets within polykey MatrixAI/Polykey#55

Closed

lheckemann mentioned this pull request Jul 16, 2020

RFC meeting 2020-07-16 NixOS/rfc-steering-committee#42

Closed

12 tasks

spacekookie mentioned this pull request Jul 30, 2020

Meeting 2020-07-30 NixOS/rfc-steering-committee#43

Closed

12 tasks

flokli mentioned this pull request Aug 6, 2020

nixos/hercules-ci-agent: init NixOS/nixpkgs#94804

Merged

10 tasks

edolstra mentioned this pull request Aug 13, 2020

Meeting 2020-08-13 NixOS/rfc-steering-committee#44

Closed

12 tasks

d-goldin closed this Aug 25, 2020

Mic92 mentioned this pull request Aug 27, 2020

Meeting 2020-08-27 NixOS/rfc-steering-committee#45

Closed

13 tasks

davidak mentioned this pull request Nov 1, 2020

Can Systemd 247 "credentials" logic solve secret management for us? NixOS/nixpkgs#102397

Closed

roberth mentioned this pull request Jan 19, 2021

nixos/systemd: allow preStart with other ExecStartPre cmdlines NixOS/nixpkgs#109976

Merged

10 tasks

matt-snider mentioned this pull request Jan 23, 2021

nixos/etesync-dav: init module NixOS/nixpkgs#83479

Merged

10 tasks

[RFC 0059]: Systemd Service Secrets #59

[RFC 0059]: Systemd Service Secrets #59

Conversation

d-goldin commented Nov 16, 2019 • edited by zimbatm

turion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

globin Nov 18, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d-goldin Nov 17, 2019 • edited

Choose a reason for hiding this comment

Ericson2314 commented Nov 18, 2019

globin commented Nov 18, 2019

globin commented Nov 18, 2019

turion commented Nov 18, 2019

aanderse commented Nov 18, 2019

globin commented Nov 18, 2019

aanderse commented Nov 18, 2019

globin commented Nov 18, 2019

aanderse commented Nov 18, 2019

d-goldin commented Nov 18, 2019 • edited

Choose a reason for hiding this comment

d-goldin Nov 20, 2019 • edited

Choose a reason for hiding this comment

d-goldin Dec 1, 2019 • edited

Choose a reason for hiding this comment

d-goldin Apr 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lassulus commented May 14, 2020

dhess commented May 14, 2020

dhess commented May 16, 2020 • edited

Notes from 2020-05-06 call

d-goldin commented May 21, 2020

dhess commented May 21, 2020

dudebout left a comment

Choose a reason for hiding this comment

Mic92 commented Jun 25, 2020

dhess commented Jun 25, 2020

aanderse commented Jul 12, 2020

Mic92 commented Jul 23, 2020

d-goldin commented Aug 2, 2020

d-goldin commented Aug 18, 2020

d-goldin commented Aug 25, 2020

Mic92 commented Aug 26, 2020

d-goldin commented Nov 16, 2019 •

edited by zimbatm

globin Nov 18, 2019 •

edited

d-goldin Nov 17, 2019 •

edited

d-goldin commented Nov 18, 2019 •

edited

d-goldin Nov 20, 2019 •

edited

d-goldin Dec 1, 2019 •

edited

d-goldin Apr 12, 2020 •

edited

dhess commented May 16, 2020 •

edited