Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Demo, RFC] Gentoo-like use-flags #56110

Closed
wants to merge 4 commits into from

Conversation

oxij
Copy link
Member

@oxij oxij commented Feb 20, 2019

What? Why?

With #55129 and following PRs merged nixpkgs can now finally have a working demo implementation of Gentoo-like use-flags of #12877. This is it (well, one of the two, see below).

(Note that I still have some leftovers to #55129, but the current state on master is 90% there already, so this can be demoed now while I have some time to write this.)

Before discussing the technical details, note that the whole point of use-flags is that they cover use cases not covered by overlays and overrides. For instance, it is very hard to maintain a working overlay of headless packages (as minimal.nix profile of NixOS dutifully demonstrates, see #29135 for more discussion), but it is almost trivial to make such a thing using use-flags.

#56109 implements "poor man's version" of use-flags by, pretty much, adding config.common option (named config.poorMansUseFlags there just for fun). It will work, but I feel that merging such an implementation would be unfortunate for several reasons:

  • it clashes with use cases of normal config.* and overlays,
  • there's no consistent naming of *Support, with*, etc options in nixpkgs,
  • using such names produces use-flags with mangled package names.

The only proper way to implement use-flags I'm aware of is to write package expressions using the following idiom:

{ stdenv, ... }:
{ pulseaudio ? false }:
stdenv.mkDerivation { ... }

(See #39580 (comment) for an explanation in more words.)

(This idiom also has an advantage of providing a way to turn those use-flags-arguments into NixOS option modules using lib.setFunctionArgs if desired, though I don't think such thing is desirable, I think option module should be defined on the top-level similarly to what #39580 wanted to do.)

Anyway, this PR adds all the infra that is needed to implement the latter, better, kind of use-flags and changes everything what needs to be changed for it to work so that the result is a noop.

The main problem there is that calling callPackage over an expression of the form

{ stdenv, ... }: argument: ...

can't be distinguished from the above idiom for use-flags automatically, so all such uses have to be patched manually.

A cleaner but currently not very likely solution is to split all-packages.nix into trivial and non-trivial parts like #50643 tried to do.

Thoughts?

(Btw, SLNOS adds a second callPackage similar to the one in this PR and uses that in all-packages.nix where needed, but such an implementation is undesirable in the long run, IMHO.)

git log

  • lib: generalize callPackageWith and callPackagesWith, add makeOverridableWithUseFlags

  • splice.nix: demo working man's use flags

  • treewide: change what needs to be changed to make this work

    In short, callPackage now can not be replaced with an import with
    some inherits, it's a bit weird w.r.t. functions.

  • mpv: make an example use-flag

nix-instantiate environment

  • Host OS: Linux 4.9, SLNOS 19.03
  • Nix: nix-env (Nix) 2.1.3
  • Multi-user: yes
  • Sandbox: yes
  • NIXPKGS_CONFIG:
{
  checkMeta = true;
  doCheckByDefault = true;
}

nix-env -qaP diffs

  • On x86_64-linux: noop
  • On aarch64-linux: noop
  • On x86_64-darwin: noop

@nixos-discourse
Copy link

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/gentoo-like-use-flags/2210/1

@danbst
Copy link
Contributor

danbst commented Feb 20, 2019

How it should be used? Also, if "headless" use-flag is introduced, would it be possible to build "headless" world by Hydra? Also, please share a list of most popular (in your opinion) useflags, be it current SLNOS ones or potential future ones.

@oxij
Copy link
Member Author

oxij commented Feb 21, 2019

How it should be used?

This infra is equivalent to "Global USE flags" of Gentoo, see https://gentoo.org/support/use-flags/. Local flags are going to be pretty easy to add after #50643, but before that it would need yet another callPackage which would take package name as a parameter.

In NixOS the above should be used similarly to

nixpkgs.config.use = {
  pulseaudio = false;
  systemd = false;
};

Also, if "headless" use-flag is introduced, would it be possible to build "headless" world by Hydra?

Yes, with some more infra similar to that was discussed in #29135. I.e. I imagine another copy of pkgs as pkgs.pkgsHeadless with headless use-flag set by default.

most popular (in your opinion) useflags

See the above link to Gentoo's page. Things like headless, pulseaudio, systemd, pam and static are popular.

@oxij oxij changed the title [Demo, RFC] Gentoo-like use-flags (will not evaluate by design) [Demo, RFC] Gentoo-like use-flags Feb 21, 2019
@oxij oxij force-pushed the demo/working-mans-use-flags branch from 99afb38 to a2e3d51 Compare February 21, 2019 05:32
@oxij
Copy link
Member Author

oxij commented Feb 21, 2019

I bit the bullet and just changed everything that needs fixing for this to become a noop for nix-env -qaP diff. I.e. now this can be used as-is with current nixpkgs. I also added an example of one use flag for mpv here for illustrative purposes. The OP was updated to reflect all of this.

In short, `callPackage` now can not be replaced with an `import` with
some `inherit`s, it's a bit weird w.r.t. functions.
@danbst
Copy link
Contributor

danbst commented Feb 21, 2019

@oxij I think Hydra relationship is the most interesting question here. Because without Hydra, such useflags can be created with help of another method: #44196

With that PR applied I can define an overlay use.nix:

self: super: let

  inherit (super) lib;

  useDefinitions = {
    pulseAudio = value: if value then {} else {
      pulseaudio = null;
      libpulseaudio = null;
      pulseSupport = false;
      pulseaudioFull = null;
      pulseaudioSupport = false;
      libpulseaudio-vanilla = null;
    };
  };

  # actually, nixpkgs.config.use, but not sure how to get it correctly in overlay
  useFlags = {
    pulseAudio = false;
  };

  packages = let
    setOfUseSets = lib.mapAttrs (n: v: useDefinitions.${n} v) useFlags;
    useSet = lib.foldl' (acc: x: acc // x) {} (builtins.attrValues setOfUseSets);
  in lib.mapAttrs (n: v:
        if builtins.isAttrs v
            then v.overrideWithScope or (x: v) useSet
            else v
    ) super;

in packages

I still need a small patch in nixpkgs to make mpv example work:

diff --git a/pkgs/development/libraries/ffmpeg/4.nix b/pkgs/development/libraries/ffmpeg/4.nix
index 9821357de3b..03481cafb11 100644
--- a/pkgs/development/libraries/ffmpeg/4.nix
+++ b/pkgs/development/libraries/ffmpeg/4.nix
@@ -1,6 +1,7 @@
 { stdenv, callPackage, fetchpatch
 # Darwin frameworks
 , Cocoa, CoreMedia
+, libpulseaudio
 , ...
 }@args:

And, finally a demo:

$ NIX_PATH=nixpkgs=$PWD nix-instantiate -A mpv
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/xsly175q6rg9my9s5rj1zi3gdywjshs2-mpv-0.29.1.drv

$ nix-store -qR /nix/store/xsly175q6rg9my9s5rj1zi3gdywjshs2-mpv-0.29.1.drv | grep pulse

$

@7c6f434c
Copy link
Member

Note that the minimal profile doesn't even use overlays. I think the problem with overlays is not so that they don't allow to achieve something, but that we don't even trust them to be good enough for use in mainline NixOS.

(Do they have an evaluation overhead or what?)

I would expect an overlay that overrides everything in xorg to null and probably overrides a bit more stuff (like gtk) to have a chance of doing the right thing, but haven't tried it.

@danbst
Copy link
Contributor

danbst commented Feb 21, 2019

@7c6f434c all stages in stages.nix are overlays. Even packageOverrides argument is massaged to become an overlay. So overlays are not insecure, rather autoloading from ~/.config/nixpkgs/overlays is questionable.

Yes, there is evaluation overhead. I'm comparing instantiating mpv without my overlay and with:

  • totalBytes 58898944 -> 74932640 (16Mb increase)

For my system (without and with overlay and patch):

  • totalBytes: 425760976 -> 473587824 (48Mb increase)

There are other problem with this overlay though: it doesn't work for gnome2 (I had to filter out all gnome2 packages), it doesn't work in some cases for nested attributes, #44196 has to be patched to not blow up on memory even more...

@7c6f434c
Copy link
Member

7c6f434c commented Feb 21, 2019 via email

@spacefrogg
Copy link
Contributor

Coming from a long-term Gentoo experience myself. I understand the wish for Gentoo-like use-flags. Especially in the context of nixpkgs, though, I don't see a particular use for the proposed idiom.

I totally agree, and I would like to stress this point, that we should strive for an agreed-upon set of use-flags that can be de-/activated on the client-side, so to speak. This alone, though, would require a rethinking of a lot of packages' build scripts to support headless/database-type-select/language-binding/whatever operation.

I don't see any use for the double function idiom. With a defined use-flags naming scheme, this could just run along callPackage supported by overlays. callPackage can do everything use-flags can, today, but it can do much more. And for the proposed approach to work you would need to have a naming scheme anyway, making the double function idiom obsolete.

What I do see as a major improvement would be to build some nixos module infrastructure that creates an ad hoc overlay that sets all relevant use-flags and the necessary library dependencies. I believe, this curated infrastructure would be necessary anyhow, because Gentoo use-flags can do something that we cannot express easily right now. Use-flags can create use-flag dependencies on other packages. I know there are assert statements, but they broke quite often in the past.

From a package implementors point of view, the main concern is, that one has to write the build scripts with the possible use flag combinations in mind. I implemented something like the following locally but never came around to polish it:

{ stdenv
, libA, libA-variantB
, pkgB, pkgB-variantB }:
stdenv.mkDerivation {
...
  buildInputs = [ libA pkgB ];
  meta = {
  ...
    validArguments = { libA = [ libA          libA-variantB libA ];
                       pkgB = [ pkgB-variantB pkgB          null ]; };
  };
}

When I implemented this scheme, I used it to automatically build a particular package in all interesting combinations (three in this case). This scheme could, of course, also be used to enforce particular dependencies.

I played with several other approaches but came to the conclusion that the information about reasonable package variants have to be kept in the package itself.

@danbst
Copy link
Contributor

danbst commented Feb 21, 2019

offtopic>

I wonder if there is any restructuring of Nixpkgs that could reduce the amount of pitfalls around overlays; waiting for Nix3 and full language support for extensible attribute sets doesn't sound like an attractive short-term plan…

One of the ways is described in #54266 (comment) - add a "guidance" function to each overlay, including all-packages, so it defines (guides) how various overlay components compose. It could be used here as well - override only leaves, recurse into sets.
/offtopic>

@oxij btw, #39580 is nice! Why didn't you want to continue in that way? I've tried it, it still had to have a fix, but small one:

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 997fbd7ed73..a2bf393c59e 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -18403,9 +18403,8 @@ in
   mpv = callPackage ../applications/video/mpv rec {
     inherit lua;
     waylandSupport     = stdenv.isLinux;
     alsaSupport        = !stdenv.isDarwin;
-    pulseSupport       = !stdenv.isDarwin;
     rubberbandSupport  = !stdenv.isDarwin;
     dvdreadSupport     = !stdenv.isDarwin;
     dvdnavSupport      = !stdenv.isDarwin;
     drmSupport         = !stdenv.isDarwin;

@grahamc
Copy link
Member

grahamc commented Feb 21, 2019

Sounds like an actual RFC might be in order.

@oxij
Copy link
Member Author

oxij commented Feb 21, 2019 via email

@oxij
Copy link
Member Author

oxij commented Feb 21, 2019

@danbst

I've tried it, it still had to have a fix, but small one:

Yep, that's what I was talking about in #39580 (comment). See the second commit of #56088 :)

@LnL7
Copy link
Member

LnL7 commented Feb 21, 2019

I'm personally not a fan of this kind of global flags because it happens to the entire tree, resulting in large permutations and unpredictable behaviour. For example this pattern isn't usable for packages that are part of the stdenv without making our cache entirely useless for the user. That said, global config options, while not prevalent, are already a thing so not everybody shares this opinion.

What would be much more interesting to me is "enable flag to all runtime dependencies of a package". Think replaceSystemDependencies/replaceDependency but for flags/options.

Maybe it's still missing, but it's not clear to me how applying flags for just one package would work with the new format and compatibility is probably also something to look into. Giving out of tree usages a transition period seems given that a significant amount of packages changed.

@zimbatm
Copy link
Member

zimbatm commented Feb 21, 2019

{ pulseaudio ? false }:

haha @oxij, the crusade against pulseaudio continues :)

@samueldr
Copy link
Member

{ pulseaudio ? false }:

haha @oxij, the crusade against pulseaudio continues :)

I'm not sure this is (entirely) related. I think this instead is to make all "use" flags logically work the same. All use are true for enabled and false for disabled. I don't see proof in the implementation, but I would assume that in the interest of making this a no-rebuild change it would default to true through flags instead of through tortuous package-dependent logic.

Even, here I'm doubting the "mpv: make an example use-flag" commit is right; the intermediary function has pulseaudio ? stdenv.isLinux, which I would rather see as pulseaudio ? false and the default use flags have pulseaudio = stdenv.isLinux or similar. Am I misinterpreting something in the demo?

@oxij
Copy link
Member Author

oxij commented Feb 22, 2019

@samueldr Yes, I think the ultimate solution is to have #56158 married with NixOS module like #39580 had. (Which I will do after I discover a good way to marry them, typing a part of config is not trivial, it turns out.) Plus split callPackage into callPackage and callWithScope, but that's of low priority.

@oxij
Copy link
Member Author

oxij commented Feb 23, 2019

Closed in favor of #56227.

@oxij oxij closed this Feb 23, 2019
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-are-optional-deps-in-nixpkgs-standardised-as-a-prominent-reference-for-pkgs-authors/30397/1

@infinisil infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Sep 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: emacs 6.topic: fetch 6.topic: GNOME GNOME desktop environment and its underlying platform 6.topic: golang 6.topic: haskell 6.topic: pantheon The Pantheon desktop environment 6.topic: python 6.topic: qt/kde 6.topic: ruby 6.topic: rust 6.topic: vim 6.topic: xfce The Xfce Desktop Environment 8.has: documentation 10.rebuild-darwin: 0 10.rebuild-linux: 0 significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet