Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib/types: Allow paths as submodule values #76861

Merged
merged 3 commits into from Jan 12, 2020

Conversation

infinisil
Copy link
Member

@infinisil infinisil commented Jan 3, 2020

Motivation for this change

Because allowing paths as submodule values isn't always backwards compatible, that change from #75031 had to be reverted.

This restores that behavior but also fixes the backwards incompatibility in nixpkgs and adds a release note about it.

Ping @roberth @danbst @grahamc

Things done
  • Updated docs with this change
  • Ran lib/tests/modules.sh successfully (again, the revert actually broke a test for this very thing in there)

@roberth
Copy link
Member

roberth commented Jan 3, 2020

Can't we solve the problem by flipping the either arguments where necessary?
In the long run parameters like this add up to a significant overhead for users and maintainers, in particular

  • documentation becomes harder to use
  • complexity of the module system logic
  • code breaking when eventually switching from submodule to submoduleWith

We should lean towards simplification and document any required changes to user modules in the release notes.

@infinisil
Copy link
Member Author

@roberth Tried that already, but unfortunately not as easy as that. Main problem is that types.path.check fails for things that can't be coerced to strings, and I don't see an easy way to fix this (we'd need some builtins.isCoercibleToString function). This is why it's in the second place in that either.

If you have a decent implementation of types.path.check then we could maybe use that instead and hope to fix all such either instances instead. But we then still can't fix either occurrences outside of nixpkgs.

@roberth
Copy link
Member

roberth commented Jan 5, 2020

Blocked on #76977 it seems. The builtin doesn't solve the other problem with path.check, which is adding intermediate cleanSourceWith sources to the store.

But we then still can't fix either occurrences outside of nixpkgs.

Can be done in the release notes. I don't expect it to be a big problem (famous last words?).
Alternatively we can document the temporary nature of the new parameter and deprecate submodule.

@infinisil infinisil mentioned this pull request Jan 6, 2020
1 task
@infinisil
Copy link
Member Author

Now that we could switch all either submodule path to either path submodule, a problem is that either submodule path won't work with future versions of nixpkgs, and either path submodule won't work with past versions of nixpkgs. This means projects defining such a type can't accommodate all different nixpkgs versions if they use such a type.

@infinisil
Copy link
Member Author

Uses are probably few though, so it might not matter much

@roberth
Copy link
Member

roberth commented Jan 7, 2020

This can be further remedied by backporting your path.check fix #77133 if necessary.

infinisil added a commit that referenced this pull request Jan 8, 2020
Previously when this function was called without a value coercible to a
string it would throw an error instead of returning false. Now it does.

As a result this now allows the use of a type like `either path attrs`
without it erroring out when a definition is an attribute set.

The warning about there not being a isPath primop was removed because
this is not the case anymore, there is builtins.isPath. But also there
always was `builtins.typeOf x == "path"` that could've been used
instead. However the path type now stands for more than just path types,
but absolute paths in general.

(cherry picked from commit d7a109b)

See #76861 (comment)
for why this is cherry-picked
@infinisil
Copy link
Member Author

Removed the allowsPathsAsValues again, fixed the either in certmgr, backported the path.check fix and added a release note for the incompatibility.

@danbst
Copy link
Contributor

danbst commented Jan 9, 2020

I'd say this is bad. Can't we already use types.submodule { imports = [ ./path-to-options ]; }? If shortcut brings such tricky behavior, maybe its not worth adding it.

@roberth
Copy link
Member

roberth commented Jan 9, 2020

@danbst It's for values, which occur more frequently than option/type declarations. (I think you mistyped?)
The real trickiness is in either, not submodule.
I think it's worth it because it makes the syntax for modules more consistent and therefore easier to understand.

@infinisil
Copy link
Member Author

It allows e.g.

some-submodule = mkMerge [
  ./some-module.nix
  (mkIf some-condition ./other-module.nix)
]

instead of

some-submodule = mkMerge [
  { imports = [ ./some-module.nix ]; }
  (mkIf some-condition { imports = [ ./other-module.nix ]; })
]

It's not that big of a benefit, but I don't think we should not allow that just because of a slight inconvenience with the either type in a very special case.

Co-Authored-By: Robert Hensing <roberth@users.noreply.github.com>
@danbst
Copy link
Contributor

danbst commented Jan 9, 2020

@infinisil I don't think it is possible to mkMerge sub modules. Will it be possible after this change?

nix-repl> (evalModules { 
  modules = [ 
   { 
    options.bla = mkOption { 
        type = with types; submodule (mkMerge [ { options.foobar = mkOption {}; } ]); 
    }; 
   }  
   { bla = {}; } 
  ]; }
).config.bla
error: The option `bla.options' defined in `<unknown-file>' does not exist.

However, if I remove mkMerge [], it works.

@danbst
Copy link
Contributor

danbst commented Jan 9, 2020

@roberth this also adds an exception, when order of items in either matters. ADT with exceptions :)

I can't build of top of my head a counterexample (when either is "ADT with exceptions" with current types), maybe it is flawed already. In that case, adding another "exception" is fine.

@roberth
Copy link
Member

roberth commented Jan 9, 2020

@danbst as much as I'd like either to be an algebraic sum type, it isn't. It's an untagged union with untagged union problems.

Also, because a mathematical union discards duplicates, if more than one field of the union can take on a single common value, it is impossible to tell from the value alone which field was last written.
-- wikipedia

either should have been called union.
"Either" is commonly used for tagged unions, in Haskell, Scala, Purescript and a bunch of languages and libraries. It should have been avoided.

@infinisil
Copy link
Member Author

infinisil commented Jan 9, 2020

Oh yeah, what I showed above (the non-path version) doesn't work, it needs to be this (if you don't use submoduleWith) which is even worse:

some-submodule = mkMerge [
  ({ ... }: { imports = [ ./some-module.nix ]; })
  (mkIf some-condition ({ ... }: { imports = [ ./other-module.nix ]; }))
]

Compare that to the version when paths are allowed

@danbst
Copy link
Contributor

danbst commented Jan 9, 2020

@infinisil still nope. I've cloned your branched and checked that paths don't work either:

[danbst@station:~/dev/nixpkgs-pr-76861]$ nix eval '(with import ./lib; (evalModules { 
  modules = [ 
   { 
    options.bla = mkOption { 
        type = with types; submodule (mkMerge [ ./dummy.nix ]); 
    }; 
   }  
   { bla = {}; } 
  ]; }
).config.bla
)'
error: value is a path while a set was expected, at /home/danbst/dev/nixpkgs-pr-76861/lib/modules.nix:208:25
(use '--show-trace' to show detailed location information)

And yes, if I remove mkMerge it works.

@infinisil
Copy link
Member Author

Ah I didn't pay attention to the code, no that doesn't work because mkMerge isn't supposed to be used for option types, it's only for the config section. Your example without mkMerge already worked previously indeed. This change is only for the cases like the example I showed (assigning submodules in the config section)

@infinisil
Copy link
Member Author

infinisil commented Jan 9, 2020

Here's a simple full example of what's now possible (Edit: Which is pretty much what the test case does)

config.nix:

{ config, lib, pkgs, ... }: {
  options.some-submodule = lib.mkOption {
    type = lib.types.submodule [];
  };

  config.some-submodule = ./some-module.nix;
}

some-module.nix:

{ lib, ... }: {
  options.foo = lib.mkOption {
    default = false;
  };
}
$ nix-instantiate --eval nixos --arg configuration ./config.nix -A config.some-submodule.foo
false

@danbst
Copy link
Contributor

danbst commented Jan 9, 2020

@roberth okay, you are right. either (attrsOf str) (attrsOf int) behaves differently depending on order of type arguments....

@infinisil
Copy link
Member Author

@roberth Good to merge?

@roberth roberth merged commit 9884cb3 into NixOS:master Jan 12, 2020
@infinisil infinisil deleted the paths-as-submodules branch January 12, 2020 13:20
arcnmx added a commit to arcnmx/nixpkgs that referenced this pull request Jan 14, 2020
This more intuitively matches `types.either` and allows paths to be
coerced to submodules again, which was inhibited by NixOS#76861
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Jan 19, 2020
This more intuitively matches `types.either` and allows paths to be
coerced to submodules again, which was inhibited by NixOS#76861

(cherry picked from commit 92b464d)
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Jan 19, 2020
Until NixOS#76861 or so is merged

(cherry picked from commit be3f887)
infinisil added a commit to infinisil/ofborg that referenced this pull request Mar 15, 2020
Without this, a big part of the lib tests aren't being done, which
previously lead to e.g. NixOS/nixpkgs#76861
And this will also be useful for checked maintainers in NixOS/nixpkgs#82461
@infinisil infinisil added the 6.topic: module system About NixOS module system internals label Mar 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants