ping @flokli @arianvp for informational purposes.

I've removed the auto-reload commit. It's just a small refactor that introduces two options now. Existing installations will remain identical.

Ping @rushmorem @LnL7

This seems like a step backwards. The sysadmin now needs to choose between security vs the benefits of NixOS.

There are a number of modules which take config files and replace dummy text with real passwords in a secure way in the systemd preStart script and this seems like it would be a good fit here too. I suggest letting the sysadmin have the best of both worlds here.

@aanderse

This seems like a step backwards. The sysadmin now needs to choose between security vs the benefits of NixOS.

Which NixOS feature or security aspect does this PR neglect?

There are a number of modules which take config files and replace dummy text with real passwords in a secure way in the systemd preStart script and this seems like it would be a good fit here too.

That's not a generic solution though. Where the password goes depends on which storage backend is chosen, so the module would have to know about all the backends.

From the description I've read this PR seems to have the sysadmin choose between security through a mutable configuration file, warning the user the module may not work as expected, or a working, declarative, reproducible configuration ... where a password is stored in plain text in the nix store.

From the description I've read this PR seems to have the sysadmin choose between security through a mutable configuration file, warning the user the module may not work as expected, or a working, declarative, reproducible configuration ... where a password is stored in plain text in the nix store.

warning the user the module may not work as expected

The only warnings are about reloading and using other contents than configText. Currently the module doesn't ever reload the service and if you don't use configText, you have an important reason not to use NixOS as "intended", presumably.

or a working, declarative, reproducible configuration

This works. It's declarative. The password comes from the deployer as is usual with NixOps. Perhaps this is what makes this solution a bit limiting? Does seem reproducible to me, certainly on NixOps.

@aanderse I guess we could ask the user to write some placeholders in the config file, have a way to map those to files and substitute those before starting? I have no idea how to do achieve that before reloading though. That's important because you want to restart vault as little as possible, to avoid downtime due to unsealing.

I guess we could use the JSON format to overlay the sensitive bits from a separate file generically. I'd implement RFC 42-style settings as well in that case.

Too bad Vault doesn't support HCL functions. hashicorp/vault#10633 would make this a non-issue.

@roberth it seems I didn't read the changes here as carefully as I should have. I believe I understand what is going on now 😆

The sysadmin can specify configPath as /run/keys/vault.hcl and nothing ends up being written to the nix store... instead nixops would end up writing the contents of the file to the target machine, including secrets, entirely without any compromise. Right? This is really smart... 💡

Thanks for your patience with me, and thanks for making a PR with something so cool in it!

@aanderse That is correct. I failed to explain that, but your comment got me thinking and digging a bit deeper, leading to #108411 which can probably replace this PR.

On the plus side what you have done here never occurred to me before, and is probably something that will useful to me. So again, thank you very much! 😃

The module system is underutilized. You can even emulate functions but it's just not obvious that it can behave like that. (Which takes a bunch of explaining 😅 )

Hi @roberth 👋

Your ideas here ended up being a bit of a pattern for me in a few nixops networks I have:

services.foo.settings = {
  # blah blah
};
deployments.keys."foo.conf" = {
  text = generators.toKeyValue { } cfg.settings;
  user = "foo";
  group = "foo";
  destDir = stateDir;
};

Secrets management can be a real hassle in NixOS. Not that I have thought it out too much, but I don't see any real problems if we were to add some sort of mechanism like deployments to NixOS itself - something that could write files out without touching the nix store.

Do you have any thoughts on that? Good idea? Bad idea? Impractical?

Any feedback appreciated. Thanks!

Hi @aanderse, I love this idea!

File copying functionality could be added to nixos-rebuild, which is the tool that has knowledge of both expressions and hosts.

To get the secret values you can either
a. run nix-instantiate first, which can return both Nixlang values that are not in the store and a derivation path for toplevel. Only requires one evaluation
b. run nix-build, then nix-instantiate in read-only mode to get the keys. This may be safer, but slower. The slowdown could be limited if you put a marker file or something in the toplevel output, indicating the presence of keys. The nix-build step needs to evaluate secrets metadata like permissions anyway, so this should be easy to add.

(Tangent: This marker file could be implemented as a flag in a generic metadata file in toplevel, something that could be shared with nixosTest's need for metadata. A marker file is more bash-friendly and therefore nixos-rebuild-friendly.)

It'd be a good idea to reuse NixOps' options, but not its implementation. Its file watching logic should be replaced and simplified by systemd units.

Also I don't think we want to have a keys group. Systemd (or any root process) can copy the keys into the desired locations with precise permissions.

For copying the files, you'll want to send an archive instead of multiple files, because scp has high per-file latency.

If you allow the (extracted?) secrets archive "creation" to be pluggable, the same interface could be supported on arion. This could probably be added later though, with a little refactoring. The various deployment tools such as Arion and NixOps can extend the options to add extra secret sources besides the "Nix-builtin" text and file ones.

Regarding NixOps we have two options:
a. Use the same option names and try to be as compatible as possible. NixOps could blacklist the NixOS module or only add its own when it detects an older NixOS.
b. Use distinct option names and allow NixOps and/or its users to transition to the NixOS module at their own pace.
I think the latter is preferable because NixOps is already undergoing plenty of change with NixOps 2.

Another interesting idea is to integrate more directly with systemd-specific techniques, such as privileged execstart(pre) lines and/or LoadCredential.

That's more thought than I thought I had. For a first iteration, you could try to keep changes to nixos-rebuild minimal and implement NixOps-style keys using systemd units to take care of the final copying from a root-private location to the various locations, setting ownership and permissions.

@roberth thanks for your feedback. That all sounds good.

ping @rnhmjoj in case they are interested in this as well... maybe? 🤷‍♂️

I'll try to find some time to create an issue for this so ideas/work could be coordinated.

Yes, I would be interested in a general mechanism to handle secrets in NixOS.

I already have worked on a minimal module that's similar to deployments.keys but uses a stable state directory (not /run/keys) and sets up the secrets with an activation script. Having something in nixos-rebuild to deploy the secrets (step that I'm currently doing by mounting a directory over sshfs) would nicely complete it.

Oh one more consideration is whether the secrets should be tied to a version of the system profile. This is required for rollbacks, but has the downside of retaining secrets for longer. It should probably be configurable. Some choices may be: keep current generation only, keep one previous generation, keep secrets for each non-GCed generation. We should also remove secrets installed by the current generation that aren't in the new generation before switching.
I guess these secrets files can be confusing. The private root-owned ones we may want to keep around for longer, whereas for the installed ones, which are potentially not root-owned, we generally want to lean towards removing, because backups for rollback are the responsibility of root.

NixOps' approach of using systemd units to indicate the availability of secrets provides more flexibility. For instance it supports secrets in ram / tmpfs and rebooting, losing the secrets. If persistent storage of secrets is chosen, installation during activation seems more robust. They're not incompatible though. The module can both provide the systemd units for the secrets and try to install the secrets during activation for the cases where that works.
@rnhmjoj's module only supports installation during activation.

NixOps' approach of using systemd units to indicate the availability of secrets provides more flexibility

I opted for an activation script because I wanted to use the module for user passwords, which are loaded during activation by the users script. How does NixOps work in this regard? I suppose you could set up the secrets in stage 1, but I'm not sure how a nixos-rebuild switch would work.

How does NixOps work in this regard?

NixOps offers some units some services so users can wait for the key files to be installed. I believe these could be better implemented as path units. That leaves the options open with regards to when the secrets are installed. NixOps installs them on nixops deploy and nixops send-keys. By default it does not provide them on startup (reboot). I'm fairly confident that NixOps rollbacks are broken wrt keys. Security may or may not be a good excuse for that.

how a nixos-rebuild switch would work.

Currently it's

1. nix-build or nix-copy-closure
2. nix-env --set (update the system profile symlink)
3. $toplevel/bin/switch-to-configuration (ie the activation scripts)

This could become

1. nix-build or nix-copy-closure
2. read system profile symlink to get generation number $i
3. copy secrets to /root/nixos-secrets/$[i+1]
4. nix-env --set
5. $toplevel/bin/switch-to-configuration $[i+1]
   1. install secrets
   2. remove old secrets (if any need to)
   3. remove /root/nixos-secrets/* entries that don't have a corresponding generation anymore

Open question: can we increment the generation number without updating toplevel; when only the secrets change? As a workaround we could hash the secrets and put them in toplevel, but this isn't great because then it can't be built without access to secrets anymore.