python: introduce config.pythonPackageOverrides for proper overrides
#40417
Conversation
The way it was done before didn't even really work because it inherited top-level `packageOverrides` into `python-packages.nix` and top-level `packageOverrides` has a different type. This patch copies the way Haskell and R do the same thing.
|
The manual explains how a `packageOverrides` (which is indeed NOT the same as one may use for the top-level) can be passed into `python`.
I see, I got confused by the fact that `python` gets called via `callPackage` with `packageOverrides` arg, my bad.
What did not work?
All I wanted is something like
```nix
nixpkgs.config.pythonPackageOverrides = self: super: {
youtube-dl = super.youtube-dl-light;
};
```
But it seems I neeed to do
```nix
nixpkgs.config.packageOverrides = pkgs: let
pyoverrides = self: super: {
youtube-dl = super.youtube-dl-light;
};
in {
<for all python versions X>
pythonX = pkgs.pythonX.override {inherit packageOverrides;};
};
```
instead.
This does completely break the current documented and used way of overriding.
I see. Well, before I made this I might have been ok with the original way, but now I think that Haskell/R way is superior.
- It's much simpler for this use case. For instance, nixpkgs refers to `youtube-dl` via several python versions, so you will have to override them all, and this style is much better for this.
- You still can override per-version in this style by matching on `isPy*`.
- You still can override without `config` as before by doing
```nix
pyPackages = pythonPackages.override {
overrides = self: super: {
foo = self.callPackage ../src/foo {};
};
};
```
like with Haskell infra.
TBH, now I feel like removing `pkgs` and moving `buildEnv` and `withPackages` `passthru`s from python derivations to `python-packages.nix` to make Python and Haskell infra match in usage completely in this regard (i.e., e.g. you would need to do `python2Packages.pythonWithPackages arg` instead of `python2.withPackages arg`).
I can fix the doc, commit messages and PR title if you feel like merging such a change. I will make the old way work again, but I intend to deprecate it.
Thoughts?
|
|
I think in general it would be good if we have a similar UI for the different package sets. But, before actually doing that I am of the opinion we should have a RFC about what this UI ought to be. And I think before that also an RFC to define the scope of Regarding the implementation. I personally dislike using
At the time I added some of these attributes ( |
|
Sorry, but when somebody says "RFC" I can only hear "please, go away". Especially when the whole change is less than ten lines of code (duplicated four times because there are four almost identical python expressions).
If you have some technical arguments against this style I'd like to hear them out. If you think we should fix Haskell infra instead, I'm would be ok with that. But if not I outlined why I prefer Haskell style above (it can express everything I can come up with and it is simpler).
I'd like to be able to do
```nix
whateverPackages.(override (self: super: {
...
})).whateverWithPackages (pkgs: with pkgs; [ foo bar ])
```
for everything I use and I'll make my branch (and very likely SLNOS) use this style regradless.
E.g. I'd like Emacs to use exactly this style too (/cc @matthewbauer and @ttuegel).
impure
Normal `nix` evaluations look at config files and hence are is impure too.
should `config` be used, or overlays?
These are orthogonal. `config` does global things (and also use-flags in SLNOS), overlays override selected derivations.
I agree that there's some intersect in `packageOverrides` and I too prefer to replace those with overlays, but those options can only be removed from `config` after all package subsets like `pythonPackages`/`haskellPackages` get a way to override for all versions of the interpreter/compiler.
E.g. I'm ok with deprecating `packageOverrides` after we have something like
```nix
pythonPackagesFor = ...
python2Packages = pythonPackagesFor python2;
```
and can override with something like
```nix
nixpkgs.overlays = [ (self: super: {
pythonPackagesFor = super.pythonPackagesFor.override {
overrides = self: super: {
foo = super.foo.override {
bar = false;
};
};
};
})]
Btw, if we rewrite the whole nixpkgs into this style then you won't really need the `config`, because you could then just `.override` the top-level package set itself with something like
```nix
nixpkgs.pkgs = (import ../default.nix {}).override {
allowUnfree = true;
};
```
|
At which point you might realize you forgot something and you want to again change your interface. Let's consider an example.
I have no technical reason against using except that, when we do a refactor, we should provide a simpler way to have multiple overrides of a sub package set, e.g. if I have an overlay A that changes
This is about the common interface. There are more package sets than Haskell and Python. We should also take those into account when designing said interface.
There you go. That is exactly the kind of thing to have an RFC for. To agree on a common direction. I'm pretty sure there will be lots of agreement so it could go fast. |
|
I cannot currently get a layer of multiple overlays overriding python-packages to work. Only one layer of the python overrides is taken into account. Would this fix that? |
|
Yes and no.
No, since `pythonPackageOverrides` is a separate thing from overlays.
Yes, since `config.packageOverrides` is actually an overlay.
Yes, because you can compose `config.pythonPackageOverrides` too.
No, because you should be able to archive the same with current master infra, just with 4x code of this infra.
Yes, for the same reason.
See my examples and the discussion of the documentation above, overriding all versions of python in an overlay on master as discussed above should work, I think.
|
|
What is the status of this pull request? |
|
This is a wontfix, I guess. Closing, though, IMHO, this is useful. |
The way it was done before didn't even really work because it inherited
top-level
packageOverridesintopython-packages.nixand top-levelpackageOverrideshas a different type.This patch copies the way Haskell and R do the same thing.