WIP: pythonPackages.virtualenv: 16.7.9 -> 20.0.21 fix #66366 #88613
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Still running |
|
@GrahamcOfBorg build pythonPackages.virtualenv python3Packages.virtualenv python38Packages.virtualenv |
andersk
reviewed
May 22, 2020
|
semi-autogenerated list with hydra links to check for failling packages
full output of nixpkgs-review(Might be, i messed up, because i had a dirty |
|
hu? stdenv is missing when building for python2Packages. |
on python2Packages at least
andersk
reviewed
May 23, 2020
calbrecht
commented
May 23, 2020
FRidh
requested changes
May 23, 2020
andersk
approved these changes
May 23, 2020
FRidh
requested changes
May 23, 2020
|
|
||
| patches = [ ./virtualenv-change-prefix.patch ]; |
There was a problem hiding this comment.
Is this patch no longer needed? if so, then the patch should also be removed from the tree.
There was a problem hiding this comment.
Actually, i am checking this right now if the comments within the patch still hold.
There was a problem hiding this comment.
oh my, it doesn't even apply. Beside that, i guess the comments are still true for python2.7 at least, somehow. The virtualenv command kind of succeeds when started with
nix-shell --pure -p 'python27.withPackages (ps: [ps.virtualenv])'
but still complains
[nix-shell:~/ws/test-66366]$ virtualenv --python python2.7 .venv
ValueError: source and destination is the same /nix/store/38kz3j1a87cq5y59k5w7k9yk4cqgc5b2-python-2.7.18/lib/python2.7/os.py
[nix-shell:~/ws/test-66366]$ python --version
Python 2.7.18
There was a problem hiding this comment.
ah, no. i am stupid. of course python2.7 is available in nix shell.
There was a problem hiding this comment.
running non-interactively with a testsetup i found the second comment of the patch
A sitecustomize.py is created in the virtualenv which makes libraries
from the python specified by the --python argument available to the
virtualenv. For example, this makes readline and sqlite3 available
when a wrapped python is specified. If no --python argument is passed,
it will only add the path to the python used when building
virtualenv, which is the unwrapped python, so sqlite3 won't be
available.
no longer holds true.
at least this is what i get from running nix-shell --pure with
shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
pythons = [
(pkgs.python37.withPackages (ps: [ps.virtualenv]))
pkgs.python35
pkgs.python36
pkgs.python38
pkgs.python39
pkgs.python27
];
in pkgs.mkShell {
nativeBuildInputs = pythons;
shellHook = ''
echo primary python is $(python --version)
for p in python3.5 python3.6 python3.8 python3.9 python2.7 ; do
virtualenv --python ''${p} --clear .venv
source .venv/bin/activate
python --version
python -c 'import sqlite3' && deactivate && echo done
done
virtualenv --clear .venv
source .venv/bin/activate
python --version
python -c 'import sqlite3' && deactivate && echo done
'';
}output
primary python is Python 3.7.7
created virtual environment CPython3.5.9.final.0-64 in 677ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.5.9
done
created virtual environment CPython3.6.10.final.0-64 in 861ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.6.10
done
created virtual environment CPython3.8.2.final.0-64 in 1044ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.8.2
done
created virtual environment CPython3.9.0.alpha.4-64 in 1204ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.9.0a4
done
created virtual environment CPython2.7.18.final.0-64 in 816ms
creator CPython2Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Python 2.7.18
done
created virtual environment CPython3.7.7.final.0-64 in 257ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.7.7
done
the first comment of the patch
Without this patch
virtualenv --python=python2.7 .fails with an
error because it notices that the python readline.so is not in the
same path as python2.7. I assume this is to avoid copying the wrong
file on systems where it is possible to find incompatible libraries by
accident. Adding "/nix/store" to the prefix fixes this problem.
still holds true, i will investigate how to overcome this
There was a problem hiding this comment.
hm, ok changing shell.nix to use only pythons withPackages, after getting closer to understanding the issue (don't know if i fully grok this yet) is giving totally different results (as a side note, i am thankful for that :)) and indeed now the python27.withPackages is totally refusing to be locked into .venv.
shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
pythons = [
(pkgs.python37.withPackages (ps: [ps.virtualenv ps.automat]))
(pkgs.python27.withPackages (ps: [ps.automat]))
(pkgs.python35.withPackages (ps: [ps.automat]))
(pkgs.python36.withPackages (ps: [ps.automat]))
(pkgs.python38.withPackages (ps: [ps.automat]))
(pkgs.python39.withPackages (ps: [ps.automat]))
];
in pkgs.mkShell {
nativeBuildInputs = pythons ++ [ pkgs.which ];
shellHook = ''
echo primary python is $(python --version)
for p in python2.7 python3.5 python3.6 python3.7 python3.8 python3.9 ; do
echo
echo Testing ''${p}
virtualenv --python ''${p} --clear .venv
source .venv/bin/activate
python --version
echo -n "import sqlite3 "
python -c 'import sqlite3' && echo succeeded || echo failed
echo -n "import automat "
python -c 'import automat' && echo succeeded || echo failed
deactivate && echo done
done
echo
echo Testing python3.7 without specifying interpreter to virtualenv
virtualenv --clear .venv
source .venv/bin/activate
python --version
echo -n "import sqlite3 "
python -c 'import sqlite3' && echo succeeded || echo failed
echo -n "import automat "
python -c 'import automat' && echo succeeded || echo failed
deactivate && echo done
'';
}output
Testing python2.7
ValueError: source and destination is the same /nix/store/38kz3j1a87cq5y59k5w7k9yk4cqgc5b2-python-2.7.18/lib/python2.7/os.py
bash: .venv/bin/activate: No such file or directory
Python 3.7.7
import sqlite3 succeeded
import automat succeeded
deactivate: command not found
Testing python3.5
created virtual environment CPython3.5.9.final.0-64 in 814ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.5.9
import sqlite3 succeeded
import automat Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named 'automat'
failed
done
Testing python3.6
created virtual environment CPython3.6.10.final.0-64 in 1094ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.6.10
import sqlite3 succeeded
import automat Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'automat'
failed
done
Testing python3.7
created virtual environment CPython3.7.7.final.0-64 in 248ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.7.7
import sqlite3 succeeded
import automat Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'automat'
failed
done
Testing python3.8
created virtual environment CPython3.8.2.final.0-64 in 1335ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.8.2
import sqlite3 succeeded
import automat Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'automat'
failed
done
Testing python3.9
created virtual environment CPython3.9.0.alpha.4-64 in 1645ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.9.0a4
import sqlite3 succeeded
import automat Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'automat'
failed
done
Testing python3.7 without specifying interpreter to virtualenv
created virtual environment CPython3.7.7.final.0-64 in 257ms
creator CPython3Posix(dest=/home/alab/ws/test-66366/.venv, clear=True, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/alab/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Python 3.7.7
import sqlite3 succeeded
import automat Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'automat'
failed
done
There was a problem hiding this comment.
@FRidh just that i get this right. With both test scenarios the second comment of the original patch about loading sqlite3 is no longer an issue. So this means loading python modules from standard library is working, right?
And, the real question i am doubting right now is, is the scond test scenario the one that is adressing the first comment of the patch, about python2.7? And furthermore, is this scenario even correct? Do we want to be able to import automat here?
There was a problem hiding this comment.
One step forward, python3 has additional base_exec_prefix and base_prefix attrs in sys. python2 does not have them. In brief explanation, they do point to the original python when in a virtual environment, and in nixos to the unwrapped python.
[nix-shell:~/ws/test-66366]$ python3.5 -c 'import sys; from pprint import pprint; pprint(vars(sys))' | grep prefix
'base_exec_prefix': '/nix/store/0df1pj661i4m292b0v82gzp0d0i8m6hf-python3-3.5.9',
'base_prefix': '/nix/store/0df1pj661i4m292b0v82gzp0d0i8m6hf-python3-3.5.9',
'exec_prefix': '/nix/store/6clz676g768xrshsrwp677nm3n49wsky-python3-3.5.9-env',
'prefix': '/nix/store/6clz676g768xrshsrwp677nm3n49wsky-python3-3.5.9-env',vs.
[nix-shell:~/ws/test-66366]$ python2.7 -c 'import sys; from pprint import pprint; pprint(vars(sys))' | grep prefix
'exec_prefix': '/nix/store/20kmxkqlh3556pgk3bawj7pc4ab7hkm2-python-2.7.18-env',
'prefix': '/nix/store/20kmxkqlh3556pgk3bawj7pc4ab7hkm2-python-2.7.18-env',There was a problem hiding this comment.
See also pkgs/development/interpreters/python/tests.nix for some tests regarding this behavior and venv.
|
@GrahamcOfBorg build pythonPackages.virtualenv python3Packages.virtualenv python38Packages.virtualenv |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation for this change
fixes #66366 in that the newer version no longer checks if
sys.prefixmatches the virtualenv.Things done
sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)before:
python2.7-virtualenv-16.7.9 102.6Mpython3.7-virtualenv-16.7.9 104.9Mafter:
python2.7-virtualenv-20.0.21 107.1Mpython3.7-virtualenv-20.0.21 108.2M