-
-
Notifications
You must be signed in to change notification settings - Fork 15.4k
Replacing modules #22764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replacing modules #22764
Conversation
@LnL7, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @ericsagnes and @nbp to be potential reviewers. |
Very good to see this! I wished this existed several times in the past. Since this mechanism is more general than replacing a module (it really just excludes a module), wouldn't it be better to substitute the name You can then optionally build a very light abstraction on top of this where the replacement is more obvious: {
replaces = [
{ old = <nixpkgs/nixos/modules/services/databases/postgresql.nix>;
new = ./postgresql.nix;
}
];
} |
I didn't really give a good example, but the idea is that a custom module can declare what modules it replaces. |
Ok you mean placing the Regardless of name, it would be cool to have this feature! |
Using a fully qualified path to denote modules is unsound because there is no guarantee that
then It might be better to have something that uses module filenames relative to the current Nixpkgs tree, e.g.
(I wouldn't call it |
I agree that it's not ideal to use the full path, but there doesn't seem to be reliable way get the path of pkgs,
|
439b645
to
66c4ddb
Compare
I think this will disable the correct modules when |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Smart trick to just use: toString ../nixos/modules + "/" + m
.
lib/modules.nix
Outdated
@@ -87,6 +84,15 @@ rec { | |||
result = { inherit options config; }; | |||
in result; | |||
|
|||
|
|||
/* Remove modules if there is another module that 'disables' it it. */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe use something like this: "Return the non-disabled modules. Disabling modules allows replacing their implementations."
lib/modules.nix
Outdated
filterModules = modules: | ||
let | ||
moduleKey = m: if isString m then toString ../nixos/modules + "/" + m else toString m; | ||
replacedKeys = map moduleKey (concatMap (m: m.disabledModules) modules); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would call replacedKeys
disabledKeys
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, that makes more sense.
66c4ddb
to
e8b3b1d
Compare
lib/modules.nix
Outdated
their implementation to be replaced. */ | ||
filterModules = modules: | ||
let | ||
moduleKey = m: if isString m then toString ../nixos/modules + "/" + m else toString m; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
../nixos/modules
suppose that all modules that would be replaced would be in NixOS repository.
I honestly does not like that because this gives option to NixOS which are not available to others.
This does not support the idea that someone else might want to use the module system for something else, such as NixUP, or extend NixOS with an additional set of default modules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's why I left in the option to use full paths, or are you referring to the nixos/modules
prefix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My point is that I do not understand why NixOS should be any different than any other consumers of the module system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you can find a way such that one can move this default path as an argument of the module system instead of as a static string in it, this might be preferable.
An example, even without leaving NixOS, is for sub-modules which are defined in different paths, should we name the complete NixOS module path within NixOS, or just name it as a relative file, as this trick is doing for NixOS modules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just found specialArgs.modulesPath
. If we used that, other module sets would get the same behaviour if they define define it there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wish you would have pinged me to push my work on github or at least give some credit back, referring to our discussion at the FOSDEM.
Apart from that, there is at least a design question which needs to be answer, which is: should we consider the closure of the excluded modules or only a single module at a time?
lib/modules.nix
Outdated
@@ -87,6 +84,16 @@ rec { | |||
result = { inherit options config; }; | |||
in result; | |||
|
|||
|
|||
/* Filter disabled modules. Modules can be disabled allowing | |||
their implementation to be replaced. */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: use #
comments, as done above.
lib/modules.nix
Outdated
filterModules = modules: | ||
let | ||
moduleKey = m: if isString m then toString ../nixos/modules + "/" + m else toString m; | ||
disabledKeys = map moduleKey (concatMap (m: m.disabledModules) modules); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Design question, before accepting this change. Do we want to exclude a single module, or do we want to exclude a single module and all the modules that are inherited by it?
|
||
options = { | ||
services.postgresql.enable = mkOption { | ||
description = "Replaced option."; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This example is not working as an option should requires more information to be listed.
<title>Replace Modules</title> | ||
|
||
<para>It's also possible to disable modules allowing the | ||
implementation to be replaced. This can be used to import a set of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you clarify what does it mean. Is that supposed to replace only all option definitions?
Note that implementation
is still used in many modules as a comment above the config
attribute, as the interface
is used above the options
attributes. The proper names are options definitions
and option declarations
, respectively for config
and options
attribute sets.
No, this is not an option, because this code solve the closure of the modules considered for the module system. So |
@nbp Sorry, I mainly wanted to get some discussion started around this because I think this would be very nice to have in 17.03. Feel free to close this and push your changes if you want. |
@LnL7 , also can you add tests cases for checking that |
6b28ef6
to
17864e3
Compare
@@ -1,4 +1,4 @@ | |||
{ lib ? import <nixpkgs/lib>, modules ? [] }: | |||
{ lib ? import ../.., modules ? [] }: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason why this used <nixpkgs/lib>
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is a legacy of when I created the test suite, to check against old tree of Nixpkgs, with -I nixpkgs=...
.
Nothing to worry about today.
17864e3
to
5a2abd5
Compare
I added some basic tests and a better example for the manual. Anything else that's missing or should be improved? The question about disabling a single file vs the entire module closure is also still open. I don't really have an opinion about that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This changes would look good with the following nits applied.
Minor documentation and test cases improvements to be fixed before merging. The documentation might require a second proof read once it is improved.
nit: add a change-log entry as well.
lib/tests/modules.sh
Outdated
@@ -99,6 +99,12 @@ checkConfigOutput 'true' "$@" ./define-enable.nix ./define-loaOfSub-if-foo-enabl | |||
checkConfigOutput 'true' "$@" ./define-enable.nix ./define-loaOfSub-foo-if-enable.nix | |||
checkConfigOutput 'true' "$@" ./define-enable.nix ./define-loaOfSub-foo-enable-if.nix | |||
|
|||
set -- config.enable ./define-enable.nix ./declare-enable.nix |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: add a one line comment to describe what these tests are for.
lib/tests/disable-module.nix
Outdated
@@ -0,0 +1 @@ | |||
{ ... }: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: git commit -a
? Remove this file.
# compatibility with the old module system. Not sure if this is | ||
# the most sensible policy. | ||
options = mergeModules prefix (reverseList closed); | ||
options = mergeModules prefix (filterModules (specialArgs.modulesPath or "") closed); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: This is adding a meaning to a name which used to be ignored by the module system. Add a paragraph to the existing specialArgs
comment to describe what is the intended meaning of modulesPath
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added a comment to the definition of evalModules
{ lib, ... }: | ||
|
||
{ | ||
disabledModules = [ ./define-enable.nix ]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Change lib/tests/modules/default.nix
to add a modulesPath
special argument, and use it to double check that we can lookup for modules as the string "define-enable.nix"
instead of a path.
|
||
<title>Replace Modules</title> | ||
|
||
<para>It's also possible to disable modules, removing the option |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Avoid starting new paragraph with It
, maybe use the passive form. I suggest reading the current documentation and try to mimic the current style to write these paragraph the same way.
<para>It's also possible to disable modules, removing the option | ||
declarations and config implementation defined within that file. | ||
This can be used to import a set of modules from another channel | ||
while keeping the rest of the system on a stable release.</para> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: This is actually a really valid use case. I suggest you add an additional 4 lines example which just does that assuming that nixos-rebuild
is used with an extra -I custom-nixos=/home/dev/nixos-custom/nixos
command line argument.
|
||
cfg = config.services.postgresql; | ||
|
||
in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: remove these extra white-lines within the let-in.
in | ||
|
||
{ | ||
disabledModules = [ "services/databases/postgresql.nix" ]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Disabling a module implies that you at least declare the equivalent set of option, otherwise you will probably have error messages because options are defined but never declared. I suggest you take a smaller existing module than postgresql
.
To answer my-self. We only want to exclude a single module instead of the closure of modules, because the list of imports of a module can refer to common modules. Counting the number of time a module is imported would be a nightmare to understand, and be a terrible interface. Having to list all modules imported by the disabled modules would be painful but would at least be easy to understand as opposed to any automatic system decision. |
d60ecf9
to
8bfa832
Compare
import a set of modules from another channel while keeping the rest | ||
of the system on a stable release.</para> | ||
<para><literal>disabledModules</literal> is a top level attribute like | ||
<literal>options</literal> and <literal>config</literal>. It |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow-up: mention imports
, as options
and config
are both valid arguments of modules.
follow-up: I would not be surprised that there is another part of the documentation which list all these special attribtues, in which case we should add disableModules
into it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's not really an overview except for the writing modules section. I could add it there, but I'm not sure if that's the best place.
{ config, lib, pkgs, ... }: | ||
|
||
{ | ||
disabledModules = [ "services/databases/postgresql.nix" ]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Add an introduction paragraph explaining each example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
This is based on a prototype Nicolas B. Pierron worked on during a discussion we had at FOSDEM.
93463b2
to
adaeaae
Compare
@nbp ❤️ Thank you for all the help with this. |
@globin , this change is marked as 17.03, we should either cherry-pick this change or move the release note to the next version. |
I think I'd prefer not including this in 17.03 if you don't have strong feelings, to give this a period of time on master first. |
I personally would really like see this in 17.03 because it enables more flexibility when using a stable release without having to maintain a fork of nixpkgs. |
Reverted by commit fcec3e1 |
re-made by commit d88721e. |
I would also prefer to see this in 17.03. In the documentation we can specify this is an experimental feature and that it's API might change in the future or that it might be removed. It's probably only going to be used by advanced users who know how to deal with changing API's. |
Note that with #76857 |
Motivation for this change
Enables replacing existing modules, making it easier to use an updated module while still using a stable release for the rest of the system. Example: https://gist.github.com/LnL7/bd05394218033cff17ec20eb93a44a4a
/cc @nbp
Things done
(nix.useSandbox on NixOS,
or option
build-use-sandbox
innix.conf
on non-NixOS)
nix-shell -p nox --run "nox-review wip"
./result/bin/
)