Python: indicate whether a derivation provides a Python module #30606

FRidh · 2017-10-20T12:44:21Z

Motivation for this change

The python-packages.nix file contains derivations that provide Python modules. Sometimes, one needs to determine whether packages provide Python modules. Currently, a hook is used for this, but the hook does not check whether the site-packages correspond to the right Python derivation.

Derivations providing Python modules should be marked with isPythonModule = python; where python is the interpreter that was used to build the module.

Most of the derivations in python-packages.nix are produced with buildPythonPackage but not all of them. Therefore, a helper function toPythonModule is provided that sets this attribute, and corrects the name as well.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

veprbl · 2017-10-20T16:19:14Z

pkgs/top-level/python-packages.nix

+  hasPythonModule = drv: (hasAttr "pythonModule" drv) && ( (getAttr "pythonModule" drv) == python);
+
+  # Recurse into a list of Python modules, returning all Python modules that are required.
+  requiredModules = drvs: filter hasPythonModule (closePropagation drvs);


Maybe name it requiredPythonModules or even recursiveFilterPythonModules. The meaning of requiredModules is hard to understand if you don't think in the context of Python modules.

veprbl · 2017-10-20T17:45:26Z

When I first tried python package using mkDerivation I was really surprised that

stdenv.mkDerivation {
  #...
  buildInputs = with pythonPackages; [ foo ];
}

is not automatically transformed into

stdenv.mkDerivation rec {
  #...
  buildInputs = with pythonPackages; [ foo ];
  pythonPath = requiredModules buildInputs;
}

Or perhaps even do this in bash. Say, we could call buildPythonPath() on $buildInputs instead of $pythonPath. I guess there must be a good reason to not do that. Is it because it can be too slow?

FRidh · 2017-10-20T18:45:08Z

@veprbl certain inputs are only needed at build or test time. Also, non-python dependencies should not be added to PYTHONPATH.

Certain things can be done in sh but I'm actually trying to get rid of that because its brittle and because you cannot guarantee that your module belongs to the interpreter for which you seek the modules.

veprbl · 2017-10-20T19:36:20Z

@FRidh Things that are only needed at build and test time go to nativeBuildInputs and checkInputs. My point was that it would be nice if things would work automagically with what is put in buildInputs like it already does for many different places in nixpkgs. As far as I understand, it usually is implemented via shell hooks, which is why I brought up shell. It can be also implemented fully in nix, like you did here, except that nix doesn't have information whether a dependence will produce "${drv}/lib/pythonXX/" path or not, so you need to introduce toPythonModule to explicitly mark each and every derivation manually (which is not necessarily a bad thing).

Also, yoda and rivet could use toPythonModule too.

teto · 2017-10-21T09:25:19Z

I had to clean up some space and disable some packages that were too long to recompile but finally ended up ok. It fixed the problem for me (1st version of the PR, you 've pushed a new version since then but I can't recompile on every push sry xD).
Also, non-python dependencies should not be added to PYTHONPATH.
couldn't buildInputs identify python packages via checking for isPython in the atttSet ?
Not sure what to do but ditching (i.e., automate the generation) pythonPath for buildInputs would make packaging easier.

FRidh · 2017-10-21T09:38:20Z

It fixed the problem for me

Good to hear.

couldn't buildInputs identify python packages via checking for isPython in the atttSet ?

Yes and no. Currently the Python interpreter comes with a setup hook which adds the site-packages folders to PYTHONPATH. The issue is that it does not distinguish between interpreter versions.

I would prefer to do most of this in Nix instead of through a setup hook. In case of our buildPythonPackage function and python.buildEnv we can do that easily. But in case of stdenv.mkDerivation we cannot.

Python libraries or modules now have an attribute `pythonModule = interpreter;` to indicate they provide Python modules for the specified `interpreter`. The package set provides the following helper functions: - hasPythonModule: Check whether a derivation provides a Python module. - requiredPythonModules: Recurse into a list of Python modules, returning all Python modules that are required. - makePythonPath: Create a PYTHONPATH from a list of Python modules. Also included in this commit is: - disabledIf: Helper function for disabling non-buildPythonPackage functions.

…ackages.nix

pythonPath is used to create the eventual wrappers. It does not recurse into the Python dependencies, which means e.g. requests doesn't haven its dependencies.

…Derivation does not use

FRidh · 2017-11-25T08:50:41Z

I've pushed this to staging.

FRidh added the 6.topic: python label Oct 20, 2017

FRidh mentioned this pull request Oct 20, 2017

system-config-printer can't find urllib3 #30596

Closed

FRidh added the 2.status: work-in-progress label Oct 20, 2017

FRidh changed the title ~~Python: pythonModule~~ Python: indicate whether a derivation provides a Python module Oct 20, 2017

veprbl reviewed Oct 20, 2017

View reviewed changes

FRidh force-pushed the pythonmodule branch 2 times, most recently from e2b3956 to e2fc437 Compare October 20, 2017 18:51

FRidh mentioned this pull request Oct 25, 2017

Python: do not map install_requires to propagatedBuildInputs #30804

Open

FRidh force-pushed the pythonmodule branch from 360f5ca to 8925b88 Compare November 11, 2017 12:55

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Nov 11, 2017

FRidh force-pushed the pythonmodule branch from 8925b88 to a872c30 Compare November 11, 2017 13:02

lukateras mentioned this pull request Nov 11, 2017

ibus: 1.5.16 -> 1.5.17, fix #31094

Merged

8 tasks

FRidh force-pushed the pythonmodule branch 3 times, most recently from 507e326 to 2b6152e Compare November 12, 2017 09:41

FRidh added 0.kind: enhancement 1.severity: mass-rebuild and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 2.status: work-in-progress labels Nov 12, 2017

FRidh force-pushed the pythonmodule branch from 2b6152e to cb79639 Compare November 12, 2017 09:48

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Nov 12, 2017

FRidh force-pushed the pythonmodule branch from cb79639 to fe7ee2e Compare November 12, 2017 10:20

FRidh force-pushed the pythonmodule branch from fe7ee2e to 23761be Compare November 12, 2017 10:21

FRidh requested review from Mic92, orivej, lsix and abbradar November 12, 2017 10:47

FRidh force-pushed the pythonmodule branch from 23761be to 81f8d73 Compare November 12, 2017 17:05

FRidh added 8 commits November 23, 2017 15:11

python.pkgs: use toPythonModule for packages defined outside python-p…

8d9f170

…ackages.nix

system-config-printer: recurse into python dependencies

9544383

pythonPath is used to create the eventual wrappers. It does not recurse into the Python dependencies, which means e.g. requests doesn't haven its dependencies.

python.pkgs.buildPythonPackage: meta.maintainers: don't add chaoflow

42eece0

python.pkgs.rivet: add bindings for rivet

985f0ce

python.pkgs.buildPythonPackage: do not pass attributes that stdenv.mk…

209ee2f

…Derivation does not use

python.pkgs.pytest: add missing setuptools dependency

7c3c2a0

python.pkgs.buildPythonPackage: some comments

80329e7

FRidh force-pushed the pythonmodule branch from 81f8d73 to 80329e7 Compare November 23, 2017 14:16

FRidh closed this Nov 25, 2017

This was referenced Nov 25, 2017

system-config-printer: fix python path #32024

Merged

python3.buildEnv: pick outputs correctly, refactor #31555

Closed

FRidh mentioned this pull request Dec 7, 2017

Mopidy is broken #32234

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: indicate whether a derivation provides a Python module #30606

Python: indicate whether a derivation provides a Python module #30606

FRidh commented Oct 20, 2017 •

edited

veprbl Oct 20, 2017 •

edited

veprbl commented Oct 20, 2017

FRidh commented Oct 20, 2017

veprbl commented Oct 20, 2017 •

edited

teto commented Oct 21, 2017

FRidh commented Oct 21, 2017

FRidh commented Nov 25, 2017

Python: indicate whether a derivation provides a Python module #30606

Python: indicate whether a derivation provides a Python module #30606

Conversation

FRidh commented Oct 20, 2017 • edited

Motivation for this change

Things done

veprbl Oct 20, 2017 • edited

Choose a reason for hiding this comment

veprbl commented Oct 20, 2017

FRidh commented Oct 20, 2017

veprbl commented Oct 20, 2017 • edited

teto commented Oct 21, 2017

FRidh commented Oct 21, 2017

FRidh commented Nov 25, 2017

FRidh commented Oct 20, 2017 •

edited

veprbl Oct 20, 2017 •

edited

veprbl commented Oct 20, 2017 •

edited