python docs: add an example for a virtualenv and pip through nix-shell #21837

Azulinho · 2017-01-12T16:03:16Z

Motivation for this change

To address:
#66

And to document what actually worked for me while dealing with different python projects that required a virtualenv and compilation of external dependencies.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-01-12T16:03:18Z

@Azulinho, thanks for your PR! By analyzing the history of the files in this pull request, we identified @FRidh, @bennofs and @abbradar to be potential reviewers.

FRidh

How to create a functioning virtualenv on Nix is a recurring question and it's good to address this in the manual.

When including this example it should also describe the limitations of this method. E.g., how are you dealing with packages that have extension types, like numpy?

How do you make sure the paths in the store remain available? Will you add a gc root? And what happens when you nix-shell again but nixpkgs has updated?

Can you use additional packages from Nix? And if so, how do you deal with dependencies between packages in the virtualenv and packages in the store? E.g., you make package a available via Nix, and install b via requirements.txt.

FRidh · 2017-01-12T16:08:11Z

doc/languages-frameworks/python.md

+
+```
+
+th import <nixpkgs> {};


2 letters missing here

FRidh · 2017-01-12T16:09:31Z

doc/languages-frameworks/python.md

+th import <nixpkgs> {};
+with pkgs.python27Packages;
+
+buildPythonPackage { 


we're not building a Python package, but an environment. I suggest you use python.buildEnv or else I think you might as well use stdenv.mkDerivation.

works for me

FRidh · 2017-01-12T16:09:56Z

doc/languages-frameworks/python.md

+    python27Packages.virtualenv
+    python27Packages.pip
+    stdenv
+    zlib ];


what's the purpose of all these packages? Are some hard dependencies, or are they just examples of packages?

my particular requirements.txt is for a twisted project which requires a few python modules containing C extensions. The packages listed are the dependencies required to build those particular python modules.

I guess I can expand that block stating which are required (like the virtualenv and pip), and which are related to dependencies of the modules listed in the requirements.txt file.

FRidh · 2017-01-12T16:10:41Z

doc/languages-frameworks/python.md

+  src = null;
+  # When used as `nix-shell --pure`
+  shellHook = ''
+  unset http_proxy


indentation of this block

FRidh · 2017-01-12T16:20:49Z

doc/languages-frameworks/python.md

+  # When used as `nix-shell --pure`
+  shellHook = ''
+  unset http_proxy
+  export GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt


Is this needed, and if so, why?
Maybe also SOURCE_DATE_EPOCH for when you want to build wheels?

my bad, those are actually only relevant to my own setup. pulling them out...
and adding SOURCE_DATE_EPOCH

Mic92 · 2017-01-18T07:37:34Z

Thanks.

Mic92 · 2017-01-18T07:38:02Z

I actually wanted to squash them together, but hit the wrong button...

CMCDragonkai · 2017-09-20T07:15:44Z

I'm always been confused here. The normal python docs mentions things like withPackages, pkgs.python35Packages.python.buildEnv.override (which do pretty much the same thing) and the only thing they do is create a python with the necessary nix python packages (there's no way to combine it with non-python dependencies?). However you don't actually need this, you can just use a normal shell.nix with all the python dependencies placed into the buildInputs of stdenv.mkDerivation and all of the python packages will be made available to the nix-shell installed python using packageHooks that change the PYTHONPATH variable.

So then the question becomes how does pip interact within a nix-shell. By default it simply doesn't work. I would prefer that pip worked similar to Node's npm or PHP's composer or even Haskell's stack, in that dependencies were installed in the current directory. Unfortunately this is not possible without some hacks into --root or --target or --prefix or --install-option which is quite complex as well.

The only way I've found that actually works is nix-shell (with stdenv.mkDerivation or python35Packages.buildPythonApplication) for system dependencies and python dependencies that I don't want to compile via pip + virtualenv and pip for all the other normal pip dependencies. But I thought that there would be a shellHook or something that would change pip to install into the local directory and thus we wouldn't need an extra virtualenv layer in between nix-shell and pip.

There's also the question of what happens when there's a nix installed python dependency like numpy and a numpy specified in the requirements.txt. When pip runs an installation, does it respect the fact that numpy already exists, or does it check the version requirements and decides to install a local copy, and if it does, does the resulting python find the virtualenv + pip installed one or the nix supplied one?

Finally there's the various python2nix projects that appears to take existing python projects and generate the necessary nix expressions. I'm not sure if any of these ones work well for creating new python packages for submission to nix in a similar vein to node2nix.

I found a way that doesn't require virtualenv. It's a combination of buildPythonApplication and usage of --prefix option in pip. See this: https://gist.github.com/CMCDragonkai/dcc1b538352624ea690d695ee1ff1528#python

Mic92 · 2017-09-20T08:08:32Z

doc/languages-frameworks/python.md

+    # the following packages are related to the dependencies of your python 
+    # project. 
+    # In this particular example the python modules listed in the 
+    # requirements.tx require the following packages to be installed locally 


- requirements.tx + requirements.txt

python docs: add an example for a virtualenv and pip through nix-shell

aa9a9dd

FRidh requested changes Jan 12, 2017

View reviewed changes

FRidh added 6.topic: python 8.has: documentation labels Jan 12, 2017

FRidh reviewed Jan 12, 2017

View reviewed changes

python docs: update block according to code review

58613a7

Mic92 merged commit a9495b3 into NixOS:master Jan 18, 2017

Azulinho mentioned this pull request Jan 19, 2017

Document how to customize modules available to virtualenv #66

Closed

Mic92 reviewed Sep 20, 2017

View reviewed changes

FRidh mentioned this pull request Jul 18, 2018

manual: "python modules using pip in a virtualenv" example don't work #43737

Open

danbst mentioned this pull request Feb 10, 2019

python doc: update virtualenv section #55265

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python docs: add an example for a virtualenv and pip through nix-shell #21837

python docs: add an example for a virtualenv and pip through nix-shell #21837

Azulinho commented Jan 12, 2017

mention-bot commented Jan 12, 2017

FRidh left a comment

FRidh Jan 12, 2017

Azulinho Jan 12, 2017

FRidh Jan 12, 2017

Azulinho Jan 12, 2017

FRidh Jan 12, 2017

Azulinho Jan 12, 2017

FRidh Jan 12, 2017

FRidh Jan 12, 2017

Azulinho Jan 12, 2017

Mic92 commented Jan 18, 2017

Mic92 commented Jan 18, 2017

CMCDragonkai commented Sep 20, 2017 •

edited

Mic92 Sep 20, 2017

python docs: add an example for a virtualenv and pip through nix-shell #21837

python docs: add an example for a virtualenv and pip through nix-shell #21837

Conversation

Azulinho commented Jan 12, 2017

Motivation for this change

Things done

mention-bot commented Jan 12, 2017

FRidh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Jan 18, 2017

Mic92 commented Jan 18, 2017

CMCDragonkai commented Sep 20, 2017 • edited

Choose a reason for hiding this comment

CMCDragonkai commented Sep 20, 2017 •

edited