This provides convenient and consistent dependency overrides. However, the main problem seems to be that there will be a conflict between python packages if the jsonschema and jupyterlab (which brings jsonschema3) will appear in the same closure.

@veprbl Yeah that is a problem (not a main one), and this PR doesn't solve this (if this can be solved at all).

The biggest reason I made this PR is to make it much more convenient to do transitive overrides, which is possible for a lot of other package sets already because of their use of makeScope.

What about the case where I need two packages, one needs jsonschema==2.* and the other needs jsonschema>=3. Won't the buildEnv still create a PYTHONPATH that has both versions on it?

Example of a shell.nix:

...
buildInputs = [
  jupyter_server # needs jsonschema_3
  jsonschema_2
];
...

@jonringer Yeah I think that would create such a conflicting env because of how buildInputs treats python packages, but if you do this it should work:

{
  buildInputs = [
    jupyterlab_server
    (python3.withPackages (p: [ p.jsonschema ]))
  ];
}

@FRidh For jupyterlab to build at all. See the PR description for what would be needed to make it build without this change.

I guess this is mostly useful for getting end-user applications to build correctly

@infinisil in applications, there's a lot more freedom to change python versions, because there's a good guarantee that that the python packages will be wrapped and isolated from other python packages. A common example is the aws-sam-cli package.
However, I think your scope example is a lot more concise than the aws-sam-cli/override example :) 👍

Hm I'm not sure I'm getting what you're saying @FRidh, but maybe it's that if I define jupyterlab in top-level/all-packages.nix, I can apply an overlay to the python package set before using callPackage, therefore achieving the same deep override as shown above, but without having to change anything in the python package set. This would mean pkgs.pythonPackages.jupyterlab would be removed in favor of pkgs.jupyterlab with custom python overrides. Is that what you mean?

I still feel like this functionality could be useful though, and it's already built into the makeScope function which is commonly used for package sets. It essentially allows adding overlays to already constructed package sets, which can make it more ergonomic in some cases, e.g. when you only receive a python package set in a function but need to add an overlay to it and you don't know where it came from.

This would mean pkgs.pythonPackages.jupyterlab would be removed in favor of pkgs.jupyterlab with custom python overrides. Is that what you mean?

Correct.

@FRidh Alright, then let's use that to get jupyterlab to build. You onboard with merging this just for the sake of uniformity of package sets in nixpkgs?

In general I am of the opinion that we need an RFC first to detail how the various package sets should look. I do not see any benefit feature-wise of this PR, although uniformity is absolutely desirable. It cannot cause any regressions either so that's good. However, the use case you had in mind cannot be dealt with in this way as discussed. By merging this change I expect more PR's with overrideScope inside the Python package set. That is something I would prefer to avoid.

Given that there's a lot of other ways to do bad things that's not a good enough reason, so I'd say go ahead with it. Just update the commit message to change the self.overrideScope' to e.g. a python3Packages.overrideScope'. We should also document that it should not be used within the Python package set.

I updated the commit message to change self -> python.pkgs (where applicable).

I wouldn't really know where to put docs for this however, and given that this functionality might be needed in some cases (maybe even in the python package set), I wouldn't want to tell people to not use it. Unless you insist on docs I'd like this to be merged so the jupyterlab update can be merged too.

The section How to override a Python package? in the Nixpkgs manual could be extended with an example using this and a warning it should not be used from within the Python packages set.

Is this just waiting on documentation?

Also, for fun (wanting to experiment with mirroring Nixpkgs's structure in a NUR repo), you can actually do this on today's Nixpkgs with a bit of backing horror:

bb010g/nur-packages@2ba91b8

Uncommented, abridged, and slightly modified code taken from 2ba91b8:pkgs/default.nix#L47-87 using #67423 as an example:

let
  # pkgs = self;
  pkgs' = let
    inherit (pkgs) lib;
    pkgsScope = lib.makeScope pkgs.newScope (_: pkgs);
    pythonScopeOverrides = self: super: {
      pythonInterpreters = super.pythonInterpreters.override (o: {
        pkgs = o.pkgs // {
          callPackage = fn: args: o.pkgs.callPackage fn (let
            packageOverrides = pySelf: pySuper:
              pySuper.__extends__ or
                (let pyScope = lib.makeScope self.newScope (self: pySuper // {
                  inherit (self) callPackage;
                }); in if args ? packageOverrides then
                  pyScope.overrideScope' args.packageOverrides
                  else
                    pyScope);

              f = if lib.isFunction fn then fn else import fn;
              in if (lib.functionArgs f) ? packageOverrides then
                args // { inherit packageOverrides; }
              else
                args);
        };
      });
      inherit (self.pythonInterpreters) python37;
      python37Packages = self.python37.pkgs;
      python3 = self.python37;
      python3Packages = self.python3.pkgs;
    };
  in pkgsScope.overrideScope' pythonScopeOverrides;
in { inherit (pkgs'.python3Packages.overrideScope' (self: super: {
  jsonschema = self.jsonschema3;
  jupyterlab = self.callPackage ../applications/editors/jupyterlab { };
  jupyterlab_server = self.callPackage ../servers/jupyterlab { };
})) jupyterlab jupyterlab_server; }

Neat and hacky!

Yeah I think the only thing missing here is docs wanted by @FRidh, but I'm frightened by xml. Imo leaving out docs for this isn't too bad, because nobody would even know they could now use .overrideScope' without looking at the source code, so no need to point out to people to not use it for python-packages.nix either. We can add docs later still.

If you don't want to point it out, why add it?

It's still useful when you need it (such as for jupyterlab and probably more cases as well), even without docs. Docs would be nice yeah, but I don't even use python or jupyterlab, so I don't really have much motivation for this.

If anybody else feels like doing this, feel free to use the patch in this PR

Agreed, the azure-cli python package list would have been a good candidate

Am bummed that this never merged; it would have been a nice solution for composable Python overrides.

Am bummed that this never merged; it would have been a nice solution for composable Python overrides.

@mikepurvis pythonPackages now uses lib.makeScopeWithSplicing, a variant of lib.makeScope that supports splicing properly. Override via .overrideScope. I currently use the following Nixpkgs overlay to overlay all Python interpreters at once via pkgsFinal: pkgsPrev: { pythonInterpreters = pkgsPrev.pythonInterpreters.overrideScope (pyFinal: pyPrev: …); }:

let
  inherit (builtins) hasAttr listToAttrs map;

  flow = f: g: x: g (f x);

  mapAttr = name: f: set: set // { ${name} = f set.${name}; };
  mapAttrOr = name: f: nul: set: if hasAttr name set then mapAttr name f set else nul;
  mapAttrOrElse = name: f: nulF: set: mapAttrOr name f (nulF name set) set;

  isOverridable = hasAttr "override";
  isScope = hasAttr "overrideScope";
in
pkgsFinal: pkgsPrev: let
  inherit (pkgsFinal.lib) composeExtensions makeOverridable;
  attachScope = f: genFinal:
    let final = genFinal final // {
      overrideScope = g: attachScope (composeExtensions f g) genFinal;
      overrideScopeGenFinal = g: attachScope f (g genFinal);
      scopeGenFinal = genFinal;
      scopeOverrides = f;
    }; in final;
  attachEmptyScope = attachScope (final: prev: { });
in {
  pythonInterpreters = let
    pyInterpsPrev = pkgsPrev.pythonInterpreters;
  in if isScope pyInterpsPrev then pyInterpsPrev else attachEmptyScope (final:
    pyInterpsPrev.override (mapAttr "pkgs" (pkgs':
      mapAttr "callPackage" (callPackage:
        fn: flow (mapAttrOrElse "overrides" (overrides:
          pkgs'.lib.composeExtensions final.scopeOverrides overrides
        ) (pyFinal: pyPrev: pyPrev)) (callPackage fn)
      ) pkgs'
    ))
  );
}