Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replacing modules #22764

Merged
merged 2 commits into from
Feb 27, 2017
Merged

Replacing modules #22764

merged 2 commits into from
Feb 27, 2017

Conversation

LnL7
Copy link
Member

@LnL7 LnL7 commented Feb 13, 2017

Motivation for this change

Enables replacing existing modules, making it easier to use an updated module while still using a stable release for the rest of the system. Example: https://gist.github.com/LnL7/bd05394218033cff17ec20eb93a44a4a

/cc @nbp

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@LnL7 LnL7 added this to the 17.03 milestone Feb 13, 2017
@mention-bot
Copy link

@LnL7, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @ericsagnes and @nbp to be potential reviewers.

@basvandijk
Copy link
Member

basvandijk commented Feb 14, 2017

Very good to see this! I wished this existed several times in the past.

Since this mechanism is more general than replacing a module (it really just excludes a module), wouldn't it be better to substitute the name replaces with excludes or disables? Whether you use it for replacing a module is up to the user.

You can then optionally build a very light abstraction on top of this where the replacement is more obvious:

{
  replaces = [
    { old = <nixpkgs/nixos/modules/services/databases/postgresql.nix>;
      new = ./postgresql.nix;
    }
  ];
}

@globin globin requested review from edolstra and nbp February 14, 2017 08:41
@LnL7
Copy link
Member Author

LnL7 commented Feb 14, 2017

I didn't really give a good example, but the idea is that a custom module can declare what modules it replaces.

@basvandijk
Copy link
Member

Ok you mean placing the replaces = [ ... ]; in ./postgresql.nix itself. Then the name makes more sense indeed. I guess I would still prefer a name which describes what operational effect it has instead of the intended application. However, I don't care that much about it.

Regardless of name, it would be cool to have this feature!

@edolstra
Copy link
Member

edolstra commented Feb 14, 2017

Using a fully qualified path to denote modules is unsound because there is no guarantee that <nixpkgs> points at the Nixpkgs tree used in the current evaluation. For example, if I do

nix-build /my/nixpkgs/nixos/tests/foo.nix -A test

then <nixpkgs> does not point at /my/nixpkgs. Similarly, in a multi-machine NixOps network spec, each machine could use a different Nixpkgs tree. (I don't think NixOps actually already supports that, but you get the idea.)

It might be better to have something that uses module filenames relative to the current Nixpkgs tree, e.g.

disableModules = [ "nixos/modules/services/databases/postgresql.nix" ];

(I wouldn't call it replaces because it doesn't actually replace a module.) The path of the current machine's Nixpkgs tree is available via config.nixpkgs.path IIRC.

@LnL7
Copy link
Member Author

LnL7 commented Feb 14, 2017

I agree that it's not ideal to use the full path, but there doesn't seem to be reliable way get the path of pkgs, config.nix.nixPath doesn't update as far as I can tell.

I also noticed this behaviour, is that expected?

$ nix-instantiate --eval -I nixpkgs=/tmp/foo -E '<nixpkgs>'  # /tmp/foo
$ nix-instantiate --eval -I nixpkgs=/tmp/foo -E '<nixpkgs/nixos>'  # /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos

@LnL7
Copy link
Member Author

LnL7 commented Feb 14, 2017

I think this will disable the correct modules when <nixpkgs> is not the active source of the nixos modules.

Copy link
Member

@basvandijk basvandijk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smart trick to just use: toString ../nixos/modules + "/" + m.

lib/modules.nix Outdated
@@ -87,6 +84,15 @@ rec {
result = { inherit options config; };
in result;


/* Remove modules if there is another module that 'disables' it it. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use something like this: "Return the non-disabled modules. Disabling modules allows replacing their implementations."

lib/modules.nix Outdated
filterModules = modules:
let
moduleKey = m: if isString m then toString ../nixos/modules + "/" + m else toString m;
replacedKeys = map moduleKey (concatMap (m: m.disabledModules) modules);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call replacedKeys disabledKeys.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, that makes more sense.

lib/modules.nix Outdated
their implementation to be replaced. */
filterModules = modules:
let
moduleKey = m: if isString m then toString ../nixos/modules + "/" + m else toString m;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

../nixos/modules suppose that all modules that would be replaced would be in NixOS repository.
I honestly does not like that because this gives option to NixOS which are not available to others.

This does not support the idea that someone else might want to use the module system for something else, such as NixUP, or extend NixOS with an additional set of default modules.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's why I left in the option to use full paths, or are you referring to the nixos/modules prefix?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is that I do not understand why NixOS should be any different than any other consumers of the module system.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can find a way such that one can move this default path as an argument of the module system instead of as a static string in it, this might be preferable.

An example, even without leaving NixOS, is for sub-modules which are defined in different paths, should we name the complete NixOS module path within NixOS, or just name it as a relative file, as this trick is doing for NixOS modules.

Copy link
Member Author

@LnL7 LnL7 Feb 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just found specialArgs.modulesPath. If we used that, other module sets would get the same behaviour if they define define it there.

Copy link
Member

@nbp nbp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish you would have pinged me to push my work on github or at least give some credit back, referring to our discussion at the FOSDEM.

Apart from that, there is at least a design question which needs to be answer, which is: should we consider the closure of the excluded modules or only a single module at a time?

lib/modules.nix Outdated
@@ -87,6 +84,16 @@ rec {
result = { inherit options config; };
in result;


/* Filter disabled modules. Modules can be disabled allowing
their implementation to be replaced. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use # comments, as done above.

lib/modules.nix Outdated
filterModules = modules:
let
moduleKey = m: if isString m then toString ../nixos/modules + "/" + m else toString m;
disabledKeys = map moduleKey (concatMap (m: m.disabledModules) modules);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design question, before accepting this change. Do we want to exclude a single module, or do we want to exclude a single module and all the modules that are inherited by it?


options = {
services.postgresql.enable = mkOption {
description = "Replaced option.";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example is not working as an option should requires more information to be listed.

<title>Replace Modules</title>

<para>It's also possible to disable modules allowing the
implementation to be replaced. This can be used to import a set of
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify what does it mean. Is that supposed to replace only all option definitions?

Note that implementation is still used in many modules as a comment above the config attribute, as the interface is used above the options attributes. The proper names are options definitions and option declarations, respectively for config and options attribute sets.

@nbp
Copy link
Member

nbp commented Feb 16, 2017

@edolstra

The path of the current machine's Nixpkgs tree is available via config.nixpkgs.path IIRC.

No, this is not an option, because this code solve the closure of the modules considered for the module system. So config.[...] is doomed to do an infinite recursion. This can only leave as a special attribute of the module system, and cannot be influenced by any mean by the evaluation of the configuration, like the imports.

@LnL7
Copy link
Member Author

LnL7 commented Feb 16, 2017

@nbp Sorry, I mainly wanted to get some discussion started around this because I think this would be very nice to have in 17.03. Feel free to close this and push your changes if you want.

@nbp
Copy link
Member

nbp commented Feb 22, 2017

@LnL7 , also can you add tests cases for checking that disabledModules allow some to declare an option which would otherwise conflict, and to check that we can remove a definition as well. There is a test directory under lib which already contains module test that you can follow.

@@ -1,4 +1,4 @@
{ lib ? import <nixpkgs/lib>, modules ? [] }:
{ lib ? import ../.., modules ? [] }:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why this used <nixpkgs/lib>?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a legacy of when I created the test suite, to check against old tree of Nixpkgs, with -I nixpkgs=....

Nothing to worry about today.

@LnL7
Copy link
Member Author

LnL7 commented Feb 22, 2017

I added some basic tests and a better example for the manual. Anything else that's missing or should be improved?
@nbp Does the implementation using modulesPath look good to you or would you prefer something more like a searchpath for modules.

The question about disabling a single file vs the entire module closure is also still open. I don't really have an opinion about that.

Copy link
Member

@nbp nbp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes would look good with the following nits applied.

Minor documentation and test cases improvements to be fixed before merging. The documentation might require a second proof read once it is improved.

nit: add a change-log entry as well.

@@ -99,6 +99,12 @@ checkConfigOutput 'true' "$@" ./define-enable.nix ./define-loaOfSub-if-foo-enabl
checkConfigOutput 'true' "$@" ./define-enable.nix ./define-loaOfSub-foo-if-enable.nix
checkConfigOutput 'true' "$@" ./define-enable.nix ./define-loaOfSub-foo-enable-if.nix

set -- config.enable ./define-enable.nix ./declare-enable.nix
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a one line comment to describe what these tests are for.

@@ -0,0 +1 @@
{ ... }:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: git commit -a ? Remove this file.

# compatibility with the old module system. Not sure if this is
# the most sensible policy.
options = mergeModules prefix (reverseList closed);
options = mergeModules prefix (filterModules (specialArgs.modulesPath or "") closed);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This is adding a meaning to a name which used to be ignored by the module system. Add a paragraph to the existing specialArgs comment to describe what is the intended meaning of modulesPath.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comment to the definition of evalModules

{ lib, ... }:

{
disabledModules = [ ./define-enable.nix ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Change lib/tests/modules/default.nix to add a modulesPath special argument, and use it to double check that we can lookup for modules as the string "define-enable.nix" instead of a path.


<title>Replace Modules</title>

<para>It's also possible to disable modules, removing the option
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Avoid starting new paragraph with It, maybe use the passive form. I suggest reading the current documentation and try to mimic the current style to write these paragraph the same way.

<para>It's also possible to disable modules, removing the option
declarations and config implementation defined within that file.
This can be used to import a set of modules from another channel
while keeping the rest of the system on a stable release.</para>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This is actually a really valid use case. I suggest you add an additional 4 lines example which just does that assuming that nixos-rebuild is used with an extra -I custom-nixos=/home/dev/nixos-custom/nixos command line argument.


cfg = config.services.postgresql;

in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove these extra white-lines within the let-in.

in

{
disabledModules = [ "services/databases/postgresql.nix" ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Disabling a module implies that you at least declare the equivalent set of option, otherwise you will probably have error messages because options are defined but never declared. I suggest you take a smaller existing module than postgresql.

@nbp
Copy link
Member

nbp commented Feb 26, 2017

Apart from that, there is at least a design question which needs to be answer, which is: should we consider the closure of the excluded modules or only a single module at a time?

To answer my-self. We only want to exclude a single module instead of the closure of modules, because the list of imports of a module can refer to common modules.

Counting the number of time a module is imported would be a nightmare to understand, and be a terrible interface.

Having to list all modules imported by the disabled modules would be painful but would at least be easy to understand as opposed to any automatic system decision.

@LnL7 LnL7 force-pushed the replacing-module branch 2 times, most recently from d60ecf9 to 8bfa832 Compare February 26, 2017 13:36
import a set of modules from another channel while keeping the rest
of the system on a stable release.</para>
<para><literal>disabledModules</literal> is a top level attribute like
<literal>options</literal> and <literal>config</literal>. It
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow-up: mention imports, as options and config are both valid arguments of modules.

follow-up: I would not be surprised that there is another part of the documentation which list all these special attribtues, in which case we should add disableModules into it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not really an overview except for the writing modules section. I could add it there, but I'm not sure if that's the best place.

{ config, lib, pkgs, ... }:

{
disabledModules = [ "services/databases/postgresql.nix" ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Add an introduction paragraph explaining each example.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

This is based on a prototype Nicolas B. Pierron worked on during a
discussion we had at FOSDEM.
@nbp nbp merged commit 37705ac into NixOS:master Feb 27, 2017
@LnL7
Copy link
Member Author

LnL7 commented Feb 27, 2017

@nbp ❤️ Thank you for all the help with this.

@nbp
Copy link
Member

nbp commented Feb 27, 2017

@globin , this change is marked as 17.03, we should either cherry-pick this change or move the release note to the next version.

@LnL7 LnL7 deleted the replacing-module branch February 27, 2017 23:17
@globin
Copy link
Member

globin commented Feb 28, 2017

I think I'd prefer not including this in 17.03 if you don't have strong feelings, to give this a period of time on master first.

@globin globin modified the milestones: 17.09, 17.03 Feb 28, 2017
@LnL7
Copy link
Member Author

LnL7 commented Feb 28, 2017

I personally would really like see this in 17.03 because it enables more flexibility when using a stable release without having to maintain a fork of nixpkgs.
That being said it's often possible to work around it in some other way, like mkForce or renaming the module from unstable.

@nbp
Copy link
Member

nbp commented Mar 3, 2017

Reverted by commit fcec3e1

@nbp
Copy link
Member

nbp commented Mar 3, 2017

re-made by commit d88721e.

@basvandijk
Copy link
Member

I would also prefer to see this in 17.03. In the documentation we can specify this is an experimental feature and that it's API might change in the future or that it might be removed. It's probably only going to be used by advanced users who know how to deal with changing API's.

@infinisil infinisil added the 6.topic: module system About "NixOS" module system internals label May 4, 2020
@infinisil
Copy link
Member

Note that with #76857 disabledModules now works recursively, meaning any imports by any disabled modules are ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants