Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0078] System-agnostic configuration file generators #78

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

7c6f434c
Copy link
Member

Extracting configuration file generation out of NixOS service modules in a more reusable way. Based on a discussion during NxCon 2020 Hackday.

@7c6f434c 7c6f434c changed the title Initial version of RFC proposal System-agnostic configuration file generators Oct 19, 2020
@7c6f434c
Copy link
Member Author

This is based on brainstorming during NixCon mainly with @svanderburg . Sander, do you accept to co-sign this as a co-author?

(Also thanks to aanderse jtojnar edolstra 0x4A6F LnL7 for discussion)

@edolstra edolstra changed the title System-agnostic configuration file generators [RFC 0078] System-agnostic configuration file generators Oct 19, 2020
@Mic92 Mic92 removed the status: new label Oct 22, 2020
@Mic92
Copy link
Member

Mic92 commented Oct 22, 2020

Anyone who wants to shepherd this RFC?

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-isnt-more-of-home-manager-merged-into-nixpkgs/6096/29

@DamienCassou
Copy link

I like this RFC, thank you!

@svanderburg
Copy link
Member

@7c6f434c sure! RFC looks great so far. I'm going to read it in detail.

There is currently a lot of code duplication between
* NixOS
* home-manager
* nix-darwin
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another target that would benefit from this are containers that need configuration independent from NixOS modules since they don't support systemd etc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True; I am not sure what publically available codebase should be referenced as an example. My real use case is not even in the list and it is closer to what you say about containers; on the other hand, highly-visible existing parallel efforts are more convenient to discuss.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Option generation is also useful for override, such as pkgs.rofi.override = { theme = <file>; }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. But the packages are shared just fine already anyway, so not a strong motivation. Wrapper packages can already be done easily enough. If implementation turns out to be a huge success we might discuss this on a case by case basis whether to move stuff to the configuration file generator library, but let's get there first.

Comment on lines +78 to +77
port = 1234;
nodeName = "localhost";
content = "Hello";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this typed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the type is provided next to the generator function.

Output:
```
{
"subdir/listen.conf" = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An example how this would be integrated into NixOS modules might be helpful.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I am not sure it can be done easily without integrating an entire example from #42 and then extending it…

listenPort = "1234";
serverGreet = "I am localhost";
};
type = …;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this type for?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the output is intended for use with an RFCs#42 implementing NixOS module, to put into (in the simplest case) services.<service>.settings. In the simplest cases one can just have a static type for the settings entry, but I guess it doesn't hurt to repeat it here and to allow, when necessary, some variation (depending on configuration format input option, I guess)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That deserves introduction and propper reasoning in the detailed section, then?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we promise to provide type specification and serialiser.

Copy link
Contributor

@blaggacao blaggacao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A great motion to factor config management, define abstract interfaces and reduce the current bundling! ❤️ I hope some of my comments prompt you to do a clarifying fixup commit.

I'd also like to see a broader perspective on config management incorporated (my stated use case is just a top-of-my-head thought, nothing duly researched)

[summary]: #summary

Provide the configuration file generation functionality separately from NixOS
as a whole, and with scoped use of the module system.
Copy link
Contributor

@blaggacao blaggacao Oct 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a "scoped use of the module system"? Maybe affording another sentence for the message would do.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scope in the sense of visibility… reworded

# Summary
[summary]: #summary

Provide the configuration file generation functionality separately from NixOS
Copy link
Contributor

@blaggacao blaggacao Oct 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to add, that in my use case, I would have an interest in that configuration management (as a shared resource) in nix would compound different APIs, like:

  • filesystem API (configuration files)
  • socket based APIs (fully dynamic service configuration)

→ It might still be useful to apply fully dynamic service configuration declaratively through nix.

Examples of such configuration APIs are:

  • network gear TCP endpoints that respond to yang or netconf
  • k8s APIs
  • ldap configuration (set of boostrapping data after handing off data management to the ldap specific UIs)

It might be out of scope for this particular RFC, though it might be worth including in future work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not even sure it is in scope for future work!

Sure, that's a good idea in general, but this RFC is about sharing something that is already written now, and in a format that has seen some use by now. Inventing on-start configuration for things without persistent configuration is a good direction of work, but it is somewhat different type of work than described here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right maybe "future work" is too normative. Would you like to add a "broader context" (or a better naming!) section at the end (as an after thought)?

functions implementing program configuration file generation. .As the design
uses a module system, putting configuration generators next to packages is also
not a perfect solution; also a configuration generator might end up generating
a set of config for a combination of packages.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the design uses a module system, putting configuration generators next to packages is also
not a perfect solution; also a configuration generator might end up generating
a set of config for a combination of packages.

→ for terseness sake, that might entirely go into the alternative section?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.

rfcs/0078-configuration-file-generators.md Outdated Show resolved Hide resolved
Comment on lines 128 to 129
Abstract generation of configuration files with package-like flat arguments and
plain text file outputs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not clear to me. Would that be something comparable to a templating approach with variable substitution?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What you say is a reasonable implementation of this abstract interface.

This RFC proposal builds upon RFCs#42 defining a configuration file abstraction
approach.

A subtree is established in the Nixpkgs repository serving as a library of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A subtree is established in the Nixpkgs repository serving as a library of
`NixOs/nixpkgs:./configs` is established serving as a library of

(or some ore appropriate folder name)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think configs is a perfect name, but maybe it is good enough, so let's see if there are other opinions. In principle, that could be left to implementation step, should only lead to a little bit of bikeshedding

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, maybe you can change it to "folder subtree" to differenciate the token from "attribute subtree" below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. Indeed an identifier conflict, thanks. Reworded a bit.

Comment on lines 56 to 57
Each such generator takes as an input a NixOS module system based «subtree»,
e.g. the values typically bound to `cfg` variable in typical modules.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NixOS module system based «subtree»

Does it take the subtree as input (as I understand it: it's path — since above sibtree referred to a path) or rather does it take the (lazily) evaluated attribute set corresponding to current config? (I'd induce the latter is what's intended here).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the value of the subtree? A possibly nested attribute set, yes. But not to entire config, just to the service.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So here you could use "attribute subtree" to differenciate from "folder subtree" above. That would help to not get confused.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, maybe indeed worth expanding.

Comment on lines +60 to +58
file names of configuration files, and the values are attribute sets of RFCs#42
settings abstraction and serialiser.
Copy link
Contributor

@blaggacao blaggacao Oct 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
file names of configuration files, and the values are attribute sets of RFCs#42
settings abstraction and serialiser.
file names of configuration files, and the values are attribute sets with one or several attributes conforming to RFCs#42 settings notation and other required metadata.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

«well-known» is factually wrong here for some of the more interesting configuration file formats.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that was my interpretation of logSettings question on RFC42.

I changed the suggestion and eliminated well-known. It should be still a little clearer that way.

Comment on lines 63 to 64
Along with the function the corresponding type specifications for input and
output modules are provided.
Copy link
Contributor

@blaggacao blaggacao Oct 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Along with the function the corresponding type specifications for input and
output modules are provided.
Along with the function definition, its corresponding input and output types are specified.

Move this up to where the detailed idea is presented. At this position, I got almost confused as if it was an afterthought.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is still detailed-design section. We define the function, and what else is provided. In a sense, it is an afterthought, or supporting scaffolding.

Copy link
Contributor

@blaggacao blaggacao Oct 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the removed non-terse interlude this notion is now close enough to it's object (the functions). I reformulated for conciseness sake.

"Provided along with the function." can mean many things (including providing when calling).

"specified along with the function definition" is much less open, though still not perfect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhat reshuffled everything in the paragraph.

approach.

A subtree is established in the Nixpkgs repository serving as a library of
functions implementing program configuration file generation. .As the design
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
functions implementing program configuration file generation. .As the design
functions implementing program configuration file generation. Alongside these functions their respective input and output types are specified. As the design

@ehmry
Copy link

ehmry commented Oct 24, 2020

I nominate myself as a shepherd. I don't have practical experience using Nix with other inits, but I am currently working on this.

@Profpatsch
Copy link
Member

I’d be interested maybe helping out when it’s in progress. Maybe a flakes-based setup will help realizing this.

Define generic service abstraction for non-`systemd` systems, possibly with a
convertor from `systemd` units.

Consider some way of referring from package to related configuration file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? (what for) → you might want to clarify to improve intelligibility here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyone interested in this future work item now is likely to be aware of the context like passthru.tests; those who are not familiar with that will wait anyway until some detailed work is started

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your response is a little bit enigmatic with regard to rhe matter of the question, but let's make some assumptions at the risk of requiring further clarification:

Do you consider here the possibility of moving service configuration still closer to the packaging itself?

If that is the case, the (new) proposed module system for packaging might come into play. And (very) early coordination with this RFC might benefit future development.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll see from feedback, but currently the RFC argues against moving configuration generators close to packages. However, we might consider developing a way of having some kind of cross-references available at evaluation time.

@06kellyjac
Copy link
Member

https://github.com/svanderburg/nix-processmgmt/ is my favourite prototype so far for abstracting services

I know the example is listed as "a minimalistic silly example" but looking at it I have no idea how the input results in the output

@7c6f434c
Copy link
Member Author

@06kellyjac

agree on nix-processmgmt, but it looks like NixOS is headed towards more reuse of upstream units, and in an RFC aiming to factor something out of NixOS-only I did not want to go too far into things inapplicable to current direction of NixOS.

I think how the input gets converted into the output is an implementation detail? I expect whatever is described before implementation will get even the interface slightly changed once implemented; I do not think specifying internals of the function implementation is a good idea. I guess at first it should be similar to how #42 is prototyped now.

@06kellyjac
Copy link
Member

@7c6f434c
I wouldn't really want to switch to a new way of defining services, only to then adopt a nix-processmgmt approach and need to do it all over again.

I'm unclear as to whether this RFC is focusing on just configuration files or also the definiton of services, both of which having different implementations across nixos, home-manager, nix-darwin as you've listed.
There are a lot of mentions of systemd in the RFC but I think the problem of configuration files and services having different implementations in different projects are two separate but related issues.
If the RFC's goal is "Extracting configuration file generation out of NixOS service modules in a more reusable way." then I think the RFC should avoid changing the actual systemd service much/at all and leave that to another RFC.


I get not wanting to define implementation details but in the output there are file names like listen.conf and content.conf. where are they defined? what determines that listen.conf is JSON and content.conf is INI? where is it defined that the input port then becomes listenPort for listen.conf?

With #42 I can see how things were before, a proposed structure, and a full example module.

@7c6f434c
Copy link
Member Author

7c6f434c commented Oct 26, 2020 via email

@deliciouslytyped
Copy link

As one data point for usecases; I recently started wanting to be able - in nix-on-droid - to start web servers for apps from f-droid that are just frontend clients. My understanding is Termux doesn't support systemd, or something,
I imagine standard sysv init scripts will work fine since they are just shell scripts.

...and then I ended up going doing a slight bit more research:
There are differences to linux: https://wiki.termux.com/wiki/Differences_from_Linux , probably most importantly no user separation, I'm not sure if you can get fake differring UIDs. Nix-on-droid also uses the proot functionality, which maybe does fake uid0: https://wiki.termux.com/wiki/PRoot

There appears to be a bit of infra for launching service scripts:
https://old.reddit.com/r/termux/comments/cs014w/termuxservices_new_package_to_control_daemons/
https://wiki.termux.com/wiki/Termux-services
https://wiki.termux.com/wiki/Termux:Boot

A bit of a todo/note to self:

  • if we ever get sufficient functionality to generate services, it would probably be nice to tell termux upstream about it / add it to their wiki

You can observe me naively running into idiosyncrasies of the proot environment in nix-community/nix-on-droid#75 (comment) . IIRC the main issues were debugging difficulties and x11 needing patches from termux to handle setuid/setgid exiting with failure because proot refuses to emulate them - so nothing immediately relevant to service generation I think.

@blaggacao
Copy link
Contributor

Potential use case: ttacon/glorious#49 / numtide/devshell#47

@edolstra edolstra added the status: open for nominations Open for shepherding team nominations label Nov 19, 2020
@@ -0,0 +1,334 @@
---
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason 0072-commonmark-docs.md is part of this RFC, could you remove it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Squashed the edits until now. Seems to have helped.

@spacekookie
Copy link
Member

@Profpatsch would you be interested in being a shepherd on this RFC?

Generally this RFC is in need for more shepherds before being able to move onwards.

@edolstra
Copy link
Member

Sorry @7c6f434c, I must have missed your attempt to set up a shepherd meeting. Should we try again? (CC @ehmry @edolstra @roberth @svanderburg)

@7c6f434c
Copy link
Member Author

Not only you, to be fair… (I am also still in doubt whether there is something to discuss that benefits from synchronicity)

@edolstra
Copy link
Member

edolstra commented Jul 8, 2021

@7c6f434c Well, a meeting can be a good way to get the discussion unstuck and decide on steps forward. So I think it would be a good idea.

@7c6f434c
Copy link
Member Author

7c6f434c commented Jul 8, 2021 via email

@edolstra
Copy link
Member

@7c6f434c Okay, thanks. I've marked the RFC as "on hold". It can be moved back to under discussion at any time, of course, when you're ready.

roberth added a commit to hercules-ci/nixpkgs that referenced this pull request May 14, 2023
A helper for module system applications.

Naming compatible with NixOS/nix#6257
To be supported by nix flake check in NixOS/nix#8332
Concept is in the spirit of a proposed module based solution to [RFC 0078 System-agnostic configuration file generators](NixOS/rfcs#78).
roberth added a commit to hercules-ci/nixpkgs that referenced this pull request May 15, 2023
A helper for module system applications.

Naming compatible with NixOS/nix#6257
To be supported by nix flake check in NixOS/nix#8332
Concept is in the spirit of a proposed module based solution to [RFC 0078 System-agnostic configuration file generators](NixOS/rfcs#78).
roberth added a commit to hercules-ci/nixpkgs that referenced this pull request Jun 13, 2023
A helper for module system applications.

Naming compatible with NixOS/nix#6257
To be supported by nix flake check in NixOS/nix#8332
Concept is in the spirit of a proposed module based solution to [RFC 0078 System-agnostic configuration file generators](NixOS/rfcs#78).
@cdmistman
Copy link

👋 it's nearly been 2 years - anybody willing to pick this RFC back up? I'm relatively new to nix but RFC 42 seems to be the "default" now (at least, in home-manager and nix-darwin), and I'd love to see something like this happen!

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-nixbsd-a-posibility/29612/19

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-nixbsd-a-posibility/29612/22

roberth added a commit to hercules-ci/nixpkgs that referenced this pull request Jul 6, 2023
A helper for module system applications.

Naming compatible with NixOS/nix#6257
To be supported by nix flake check in NixOS/nix#8332
Concept is in the spirit of a proposed module based solution to [RFC 0078 System-agnostic configuration file generators](NixOS/rfcs#78).
@ShamrockLee
Copy link

Maybe we could start by providing information about the system (e.g. moduleSystemName) or the properties of the system (e.g. isPrivileged) through config. That would make it possible to share modules between NixOS and Home Manager, as lot of HM modules are basically NixOS modules without privileged configuration.

@kevincox
Copy link
Contributor

kevincox commented Sep 6, 2023

We don't use the On Hold label anymore, marking as draft.

@kevincox kevincox marked this pull request as draft September 6, 2023 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet