Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib/types: add the selectorFunction type #44601

Closed
wants to merge 1 commit into from

Conversation

basvandijk
Copy link
Member

@basvandijk basvandijk commented Aug 7, 2018

This came out of #38698 (comment).

This is used to type options like: services.xserver.windowManager.xmonad.extraPackages that specify functions that take an attribute set containing packages / plugins and return a list containing a selection of the values in this set.

Motivation for this change

The reason we need a dedicated type for this is to define the correct merge behaviour. Without the selectorFunction type merging multiple selector function option definitions results in an evaluation error.
The selectorFunction merges definitions by returning a new selector function that applies the selector functions of all the definitions to the given input and concatenates the results.

Bikeshedding

Instead of naming it selectorFunction we could name it selector which is shorter. However I opted for the former because it makes it immediately clear the value should be a function.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

This is used to type options like:

  services.xserver.windowManager.xmonad.extraPackages

that specify functions that take an attribute set containing packages
/ plugins and return a list containing a selection of the values in
this set.

The reason we need a dedicated type for this is to define the correct
merge behaviour. Without the selectorFunction type merging multiple
selector function option definitions results in an evaluation error.
The selectorFunction merges definitions by returning a new selector
function that applies the selector functions of all the definitions to
the given input and concatenates the results.
@infinisil
Copy link
Member

Ping @Profpatsch @rycee

This looks good to me, neat!

@oxij
Copy link
Member

oxij commented Aug 7, 2018 via email

@basvandijk
Copy link
Member Author

basvandijk commented Aug 7, 2018

@oxij great suggestion. I created #44626 as an alternative.

@nbp
Copy link
Member

nbp commented Aug 7, 2018

No, we should avoid having function as NixOS option. While this might make our life simpler to think in terms of functions, not everybody has a degree in functional programming, and we should not require as much from our users. If you disagree, you are free to have your own modules outside of NixOS, or to use one of the following suggestions:

For all the uses cases mentioned here, a simpler alternative is to add an overlay to add the set of packages out of which compatible packages can be extracted, such as pkgs.ihaskellPkgs.* or pkgs.xmonadPkgs.*. pkgs is the attribute set from which users are expected to take packages out of, not any argument given to any option as argument. Arguments of function are not extensible with the module system. On the other hand, pkgs packages are modular, to the extend that overlays are ordered properly.

Another option, is to create an attribute set which contains the packages to be used as argument, like we do with system.build (https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/activation/top-level.nix#L144). However, such attribute set as the same limitation as function argument as not being extensible.

Sorry, if this answer is not what you had expected, but until I see these alternative fail (i-e not being as expressive or more), I will consider this answer to be final.

@nbp nbp closed this Aug 7, 2018
@Profpatsch
Copy link
Member

Profpatsch commented Aug 7, 2018

@oxij is right that this is a special case of (one sensible?) function instance for Monoid, that is “a function which takes some a and returns an m that is a Monoid and can be merged”:

default :: Monoid m => a -> m
default = const default
merge :: Monoid m => (a -> m) -> (a -> m) -> (a -> m)
merge f g x = merge (f x) (g x)

In our case a -> m is Attrs a -> List a, which specializes the above definition in two dimensions: 1) the input can only be an attribute set and 2) the output list can only contain the same elements as the input set.
We could think of a lot more generalizations of a -> m; by definition a can be of any typy and m can be any of type that exists in types.nix, because they all implement merge and default and are therefore Monoids by construction.

You can easily achieve that generality in your definition as well, by giving selectorFunction an argument type t that specifies the output type and then copying how e.g. coercedTo defines their merge function (combining the arguments with t.merge instead of concatMap).

Update: lol, @nbp closes this PR as being too confusing for potential users while I’m proposing to make it more general first.

@nbp
Copy link
Member

nbp commented Aug 7, 2018

@Profpatsch Honestly, if we have to explain category theory to understand NixOS, then this project is a failure. I am glad that people can have satisfaction in the sanity of the idea using these. However, this does mean that we should live up to everything which can be expressed in this system, especially if this leads to a documentation hell.

Today, I think that functions are a documentation hell to explain to someone who has only manipulated apt-get install commands so far. So, if the alternative are disproved after being tested, then we can re-open this discussion.

@oxij
Copy link
Member

oxij commented Aug 8, 2018 via email

@basvandijk
Copy link
Member Author

basvandijk commented Aug 8, 2018

@nbp regarding your first suggestion:

a simpler alternative is to add an overlay to add the set of packages out of which compatible packages can be extracted, such as pkgs.ihaskellPkgs.* or pkgs.xmonadPkgs.*

The problem with this approach is that you make the user responsible for choosing the correct package set which is error prone. For example say a users defines the following configuration:

{ pkgs, ... }: {
   services.postgresql.package = pkgs.postgresql10;
   services.postgresql.extraPlugins = [ pkgs.postgis ];
}

This will crash the PostgreSQL server because the postgis extension is compiled against a version of postgresql which is different than postgresql10. Currently our users are supposed to fix this by overriding the extension with the PostgreSQL package in use:

{ pkgs, ... }: {
   services.postgresql.package = pkgs.postgresql10;
   services.postgresql.extraPlugins = [ 
    (pkgs.postgis.override { postgresql = pkgs.postgresql10; }) 
  ];
}

We shouldn't require our users to go to this trouble. In #38698 @thoughtpolice proposes to fix this by having users specify the following:

{ pkgs, ... }: {
   services.postgresql.packages = pkgs.postgresql10Packages;
   services.postgresql.plugins = ps: [ ps.postgis ];
}

The postgresql module will then take care of applying the plugins function to the choosen postgresql package set. This approach is more safe and it will become extensible by typing the option with selectorFunction or functionTo.

Difficulty argument

Regarding your argument that functions are too hard to use for regular NixOS users, aren't we already requiring that users understand functions by having them write configurations like:

{ pkgs, ... }: { ... }

Also having good documentation helps a lot with overcoming this problem. IMHO @thoughtpolice did a good job with this in the documentation of services.postgresql.plugins.

Alternative

A potential alternative which doesn't use functions can look like this:

{ pkgs, ... }: {
   services.postgresql.packages = pkgs.postgresql10Packages;
   services.postgresql.plugins = [ "postgis" ];
}

So instead of using a function that selects a list of attributes we use a list of strings instead. The strings will denote the names of attributes which the module will then select out of the correct package set.

In one way this will be safer since we can't accidentally return an attribute from a set different than the input set. Note that we make this error in the xmonad test.

I do wonder what kind of error messages users will get when they misspell an attribute name:
[ "postgresqk" ] and if this is worse than the error we get when using a function:
ps: [ ps.postgresqk ]...

@nbp @thoughtpolice do you see any problems with this approach?

@nbp
Copy link
Member

nbp commented Aug 15, 2018

@basvandijk I understand the problem. Thanks for explaining.
The alternative is appealing to me, but list are known to not be modular, and I was wondering wether we could use an (free-form) attribute set of booleans to toggle plugins.

{ pkgs, ... }: {
   services.postgresql.packages = pkgs.postgresql10Packages;
   services.postgresql.plugins.postgis = true;
}

Then we can add a module assertion that all of the attributes provided that way are listed in the list of associated packages.

I think that we can provide a better error message than what Nix does by default when looking up attribute sets. Thus, not using ps.postgris should help us improve the error messages friendliness.

@basvandijk
Copy link
Member Author

@nbp I like your suggestion. An additional advantage of having an explicit option for each plugin is that we can use it to add additional options per plugin. So we could use a scheme like for example:

{ pkgs, ... }: {
   services.postgresql.packages = pkgs.postgresql10Packages;
   services.postgresql.plugins.postgis = {
     enable = true;
     ... additional postgis options ...
   };
}

@thoughtpolice would this work for you?

@thoughtpolice
Copy link
Member

thoughtpolice commented Aug 17, 2018

It probably would, and I think this change can basically be isolated within the module, but there are some things to think about. I guess this is the only way you can make better error messages work (because you can throw an error based on passthru.versionCheck), but I somewhat hate bloating up the module system (and manual) with options like this if possible. And frankly I'm just not as sympathetic as other people are to concerns about "user experience for error messages" for things like this, because our user experience is simply godawful already and this, among a million trillion other things, is just a drop in an (ever increasing) bucket and relatively easily explained, at that (you'll spend 500x times longer just explaining the poor, conflicting UX of nix-* binaries themselves, not to mention things that are trivial in other distros like "disk encryption setup"). Modularity is arguably a problem, but the module system has massive problems, including being "non modular" to begin with; good luck starting two instances of Postgres, for instance.

To be clear I'm not, like, put off by that position -- I just do not find it convincing. (It was already bad, the current selectionFunction design I had just doesn't make it worse in any meaningful dimension, and improves it in several). But I would be willing to make the change if it seems to work out OK, because anything is an improvement over what we have, and I'm not exactly married to selectorFunction

I don't think the ability to specify per-plugin options is super useful right now, but maybe in the future some other code could make use of it. It might be most useful for overrides (i.e. "I want to apply this patch to my Postgis plugin"), but probably not for most configuration ("I need to set shared_preload_libraries in postgresql.conf").

@nbp
Copy link
Member

nbp commented Aug 18, 2018

@thoughtpolice bloating up the module system should not be an issue. What would be an issue is users not being to figure out that NixOS support the feature they want because of the lack of documentation. For example that you can easily create mediawiki instances: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-servers/apache-httpd/mediawiki.nix

the module system has massive problems, including being "non modular" to begin with; good luck starting two instances of Postgres, for instance.

Do not blame the module system for not being modular by the fact that NixOS does not have the features you are looking for. I bet one can write a modular-postgres.nix module and use it in NixOS.

@infinisil
Copy link
Member

We can discuss whether having function options in NixOS is a good thing all day long. But fact is, we currently do have such options, and right now they all throw an obscure error when you assign them multiple times. Because we can't just remove all those options, we need to fix this. And the best way to fix this is to have such a type as in this PR.

So I propose to add a warning such as

# Function types should only be used when it really makes sense

above the type and merge this PR. I've just ran into the same problem with @Jomik on IRC because of this.

@danbst
Copy link
Contributor

danbst commented Mar 7, 2019

I don't get "serializability" argument. As of now we can't get plain JSON out of configuration

$ nix-instantiate --eval -E 'with import <nixpkgs/nixos> { configuration.boot.isContainer = true; }; 
      config' --strict
trace: lib.zip is deprecated, use lib.zipAttrsWith instead
trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
trace: types.optionSet is deprecated; use types.submodule instead
trace: `types.list` is deprecated; use `types.listOf` instead
trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
trace: WARNING: `stdenv.isArm` is deprecated after 18.03. Please use `stdenv.isAarch32` instead
error: enum-0.4.7 not supported for interpreter python3.7
(use '--show-trace' to show detailed location information)
$ nix-instantiate --eval -E 'with import <nixpkgs/nixos> { configuration.boot.isContainer = true; }; 
        config.services' --strict
error: The option `services.networking.websockify.sslCert' is used but not defined.
(use '--show-trace' to show detailed location information)
$ nix-instantiate --eval -E 'with import <nixpkgs/nixos> { configuration.boot.isContainer = true; }; 
      config.assertions' --strict
error: attribute 'cycle' missing, at /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos/modules/tasks/filesystems.nix:214:119
(use '--show-trace' to show detailed location information)

We can't serialize derivations, so need literalExample hack. So adding a function type wouldn't make things worse.

As for selector function, I'd like to have 2 variants: types.listSelector and types.attrsSelector. Both represent a function, that takes an attrset input parameter, but produce list and attrs respectively. Mabe we can make it parametrized type (types.listSelector pkgs), but I'm not sure it is possible. The postgresql example would look then like:

options.pg = mkOption { default = pkgs.postgresql; };
options.plugins = mkOption {
  type = types.listSelector config.pg.pkgs;
  literalExample = ''pkgs: [ pkgs.pg_repack ]'';
  default = _: [];
};
config = {
  ... "${config.pg.withPackages config.plugins}/bin/postgres ....
};

Actually, I like nixsap style more:

options.plugins = mkOption {
  type = types.attrsSelector config.pg.pkgs;
  literalExample = ''pkgs: { inherit (pkgs) pg_repack; }'';
  default = _: {};
};
config = {
  ... "${config.pg.withAttrPackages config.plugins}/bin/postgres ....
};

PS: this was already started by @nbp 8 years ago

mergeConfig = lhs_: rhs_:
let
lhs = optCall lhs_ { inherit pkgs; };
rhs = optCall rhs_ { inherit pkgs; };
in
lhs // rhs //
optionalAttrs (lhs ? packageOverrides) {
packageOverrides = pkgs:
optCall lhs.packageOverrides pkgs //
optCall (attrByPath ["packageOverrides"] ({}) rhs) pkgs;
} //
optionalAttrs (lhs ? perlPackageOverrides) {
perlPackageOverrides = pkgs:
optCall lhs.perlPackageOverrides pkgs //
optCall (attrByPath ["perlPackageOverrides"] ({}) rhs) pkgs;
};
configType = mkOptionType {
name = "nixpkgs-config";
description = "nixpkgs config";
check = x:
let traceXIfNot = c:
if c x then true
else lib.traceSeqN 1 x false;
in traceXIfNot isConfig;
merge = args: fold (def: mergeConfig def.value) {};
};

@basvandijk
Copy link
Member Author

@danbst regarding:

I'd like to have 2 variants: types.listSelector and types.attrsSelector

We could also reconsider #44626 which provides functionTo which is a generalisation of selectorFunction. With that we can have:

  • listSelector = xs: functionTo (listOf xs)
  • attrsSelector = attrs: functionTo (attrsOf attrs)

@danbst
Copy link
Contributor

danbst commented Mar 8, 2019

@basvandijk that's fine. What I've proposed is allow dependent types (muahaha) in module system. Then, if we do something like this:

      plugins = mkOption {
        type = listSelector cfg.package.pkgs;
        description = "bla";
      };

we can infer argument type from cfg.package.pkgs. I recall @ts468 implemented such infer function before.

Added benefit is that we can introspect that set, for example, in future interactive configurator:

$ nix-instantiate --eval ./nixos --arg configuration '{
     boot.isContainer = true; 
     services.postgresql.enable = true;
     services.postgresql.plugins = ps: with ps; [ pg_repack ];

}' -A options.services.postgresql.plugins.type.value -A options.services.postgresql.plugins.type.arg1

<LAMBDA>
{ cstore_fdw = <CODE>; pg_cron = <CODE>; pg_hll = <CODE>; pg_repack = <CODE>; pg_similarity = <CODE>; pg_topn = <CODE>; pgjwt = <CODE>; pgroonga = <CODE>; pgtap = <CODE>; plv8 = <CODE>; postgis = <CODE>; timescaledb = <CODE>; tsearch_extras = <CODE>; }

@turion
Copy link
Contributor

turion commented Jan 12, 2021

I recently implemented something like this ad hoc in #109035. I believe it should be in lib/types.nix. Is there any progress on this?

@Profpatsch
Copy link
Member

Pinging @infinisil, since he has the best overview of the types setup right now.

@infinisil infinisil mentioned this pull request Jan 24, 2021
10 tasks
@infinisil
Copy link
Member

See #110707

@stale
Copy link

stale bot commented Jul 23, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 23, 2021
@infinisil
Copy link
Member

This has been implemented with #110707!

@infinisil infinisil closed this Mar 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants