Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackagesExtensions: override all Python package sets at once #91850

Merged
merged 1 commit into from Aug 6, 2022

Conversation

FRidh
Copy link
Member

@FRidh FRidh commented Jun 30, 2020

Python package sets can be overridden by overriding an interpreter
and passing in packageOverrides = self: super: {...};. This is fine
in case you need a single interpreter, however, it does not help you
when you want to override all sets.

With this change it is possible to override all sets at once by
appending a list of "extensions" to pythonPackagesExtensions.

From reading the implementation you might wonder why a list is used, and
not
lib.composeExtensions? The reason is the latter requires knowledge of
the library function. This approach should be easier for most users
as it is similar to how we append to lists of e.g. inputs or patches
when overriding a derivation.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@FRidh FRidh requested a review from jonringer as a code owner June 30, 2020 15:26
@FRidh FRidh requested review from infinisil and alyssais June 30, 2020 15:27
@FRidh
Copy link
Member Author

FRidh commented Jun 30, 2020

I would still prefer to go via an RFC but I don't see that happening any time soon really.

@infinisil
Copy link
Member

infinisil commented Jun 30, 2020

On one hand, I'd rather not have yet another override mechanism but make an RFC about it instead.
On the other hand, we already have so many override mechanisms, one more won't hurt ¯\(ツ)

So eh, I'm about 50/50 on doing this.

@stale
Copy link

stale bot commented Dec 29, 2020

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Dec 29, 2020
@r-burns
Copy link
Contributor

r-burns commented Jan 10, 2021

Still important to me. Without something like this, it's impossible to share python package overlays with other people in a configuration-agnostic way.

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 10, 2021
@FRidh FRidh self-assigned this Mar 13, 2021
@FRidh FRidh added this to the 21.05 milestone Mar 13, 2021
@FRidh FRidh force-pushed the pythonpackagesoverrides branch 2 times, most recently from dd44da0 to c4e9aef Compare April 24, 2021 08:45
@FRidh FRidh changed the title pythonPackagesOverrides: override all Python package sets at once pythonPackagesOverlays: override all Python package sets at once Apr 24, 2021
@FRidh
Copy link
Member Author

FRidh commented Apr 24, 2021

On one hand, I'd rather not have yet another override mechanism but make an RFC about it instead.
On the other hand, we already have so many override mechanisms, one more won't hurt ¯_(ツ)_/¯

Not just in mechanisms, but also in names! Overlays, extensions, overrides, ...

I picked this up again because now with flakes being able to override several times becomes even more important. As mentioned before, ideally we go through a RFC (e.g. NixOS/rfcs#83) and unify things more, but I just don't find the time to work on that.

@infinisil
Copy link
Member

Coming back to this, because I kind of need something like this at the moment! Because currently even if you try to overlay all packages via python39Packages = python39Packages.override { overrides = ... }, you'll still miss python39.pkgs and others. So something like this really feels necessary.

Seems like many other people agree too, I'm willing to merge if conflicts are fixed :)

@FRidh

This comment was marked as duplicate.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/add-python-package-via-overlay/19783/2

@FRidh FRidh changed the title pythonPackagesOverlays: override all Python package sets at once pythonPackagesExtensions: override all Python package sets at once Jul 20, 2022
@FRidh
Copy link
Member Author

FRidh commented Jul 20, 2022

I've rebased this commit. Also renamed overlays to extensions, to avoid confusion with Nixpkgs-wide overlays.

Python package sets can be overridden by overriding an interpreter
and passing in `packageOverrides = self: super: {...};`. This is fine
in case you need a single interpreter, however, it does not help you
when you want to override all sets.

With this change it is possible to override all sets at once by
appending a list of "extensions" to `pythonPackagesExtensions`.

From reading the implementation you might wonder why a list is used, and
not
`lib.composeExtensions`? The reason is the latter requires knowledge of
the library function. This approach should be easier for most users
as it is similar to how we append to lists of e.g. inputs or patches
when overriding a derivation.
@FRidh FRidh merged commit 2270b66 into NixOS:master Aug 6, 2022
@FRidh FRidh deleted the pythonpackagesoverrides branch August 6, 2022 07:39
@knedlsepp
Copy link
Member

Lovely! ❤️

rrbutani added a commit to rrbutani/iree-nixpkgs that referenced this pull request Nov 16, 2022
rrbutani added a commit to rrbutani/iree-nixpkgs that referenced this pull request Nov 16, 2022
# Function that when called
# - imports python-packages.nix
# - adds spliced package sets to the package set
# - applies overrides from `packageOverrides` and `pythonPackagesOverlays`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment uses the previous name, now pythonPackagesExtensions

@@ -133,6 +134,17 @@ let
# test-overrideScope = let
# myPackages = python.pkgs.overrideScope extension;
# in assert myPackages.foobar == myPackages.numpy; myPackages.python.withPackages(ps: with ps; [ foobar ]);
} // lib.optionalAttrs (python ? pythonAttr) {
# Test applying overrides using pythonPackagesOverlays.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants