Python: remove $out/bin/__pycache__ in fixup phase #30915
Conversation
|
These files could be avoided in the first place by creating an environment variable called Maybe this is a better fix? |
|
My concern with that is that we actually want Happy to do it with an environment variable though, if that's not a concern. |
|
I think the env var approach should be fine as there are different abstractions for libraries and applications ( |
No, that won't be sufficient. All Python packages can provide executables. The reason @casey opened this is exactly because of certain libraries that distribute executables ending with |
| @@ -29,4 +29,8 @@ attrs // { | |||
|
|
|||
| runHook postInstall | |||
| ''; | |||
|
|
|||
| fixupPhase = attrs.fixupPhase or '' | |||
| rm -rf $out/bin/__pycache__ | |||
There was a problem hiding this comment.
Bytecode produces by 2.x and 3.x is different.
$ ls -Al $(nix-build -A python2.pkgs.docutils)/bin
total 132
-r-xr-xr-x 1 root root 483 Jan 1 1970 rst2html5.py
-r--r--r-- 1 root root 745 Jan 1 1970 rst2html5.pyc
-r-xr-xr-x 1 root root 1620 Jan 1 1970 .rst2html5.py-wrapped
-r-xr-xr-x 1 root root 482 Jan 1 1970 rst2html.py
-r--r--r-- 1 root root 619 Jan 1 1970 rst2html.pyc
-r-xr-xr-x 1 root root 1073 Jan 1 1970 .rst2html.py-wrapped
-r-xr-xr-x 1 root root 483 Jan 1 1970 rst2latex.py
-r--r--r-- 1 root root 749 Jan 1 1970 rst2latex.pyc
...
$ ls -Al $(nix-build -A python3.pkgs.docutils)/bin
total 92
dr-xr-xr-x 2 root root 4096 Jan 1 1970 __pycache__
-r-xr-xr-x 1 root root 483 Jan 1 1970 rst2html5.py
-r-xr-xr-x 1 root root 1621 Jan 1 1970 .rst2html5.py-wrapped
-r-xr-xr-x 1 root root 482 Jan 1 1970 rst2html.py
-r-xr-xr-x 1 root root 1074 Jan 1 1970 .rst2html.py-wrapped
-r-xr-xr-x 1 root root 483 Jan 1 1970 rst2latex.py
...
Both cases need to be handled.
Then, there is the matter of where to put this function. Whoever uses buildPythonPackage, should be able to use this. Even with format = "other";. Therefore, I suggest putting this in the postFixup of mk-python-derivation.nix.
|
It would also be good to consider #29086. In that issue the suggestion is made that all of our language-specific builders should be more modular, so you could just pick parts. |
|
@GrahamcOfBorg python3Packages.requests |
There was a problem hiding this comment.
from _pytest.config import (
File "/nix/store/i2dm10k5b1qch07pgjfmq8hkpqbsihin-python3.6-pytest-3.2.3/lib/python3.6/site-packages/_pytest/config.py", line 9, in <module>
import py
ModuleNotFoundError: No module named 'py'
builder for ‘/nix/store/rw83k17p9c3cp1ya7vs5rbl9vx9z02hi-python3.6-cffi-1.11.2.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/6g25mvsb9n88z8mhrsabhjrs15ws2pw3-python3.6-cryptography-2.0.3.drv’: 5 dependencies couldn't be built
cannot build derivation ‘/nix/store/dh42gky3kpyj6frgvf1jfc7670qfbfp9-python3.6-pyOpenSSL-17.2.0.drv’: 2 dependencies couldn't be built
cannot build derivation ‘/nix/store/99ygxj0sfinc5scx7iyzz9pkikvd9814-python3.6-urllib3-1.22.drv’: 2 dependencies couldn't be built
cannot build derivation ‘/nix/store/cw4jmxc1q6svlms210qjaxrjjynw28mw-python3.6-requests-2.18.4.drv’: 2 dependencies couldn't be built
error: build of ‘/nix/store/cw4jmxc1q6svlms210qjaxrjjynw28mw-python3.6-requests-2.18.4.drv’ failed
|
It fails to build because this |
|
@FRidh If I put this in |
|
As I wrote before it needs to go in |
|
Unfortunately I'm going traveling for a few months, and my nix box is now languishing in a self-storage unit, so I won't be able to work on this until I get back. I'm going to close it. Hopefully I get a chance to re-open it, or some kind soul will be able to pick it up before I get the chance. |
Motivation for this change
This fixes #29896.
Things done
build-use-sandboxinnix.confon non-NixOS)nix-shell -p nox --run "nox-review wip"./result/bin/)Tested by creating a minimal
configuration.nixwhich, when built, producedresult/sw/bin/__pycache__.Then rebased this diff on top of it and rebuilt, and
__pycache__was gone.I'm slightly concerned about testing here. This seems like a safe diff, but it potentially effects every python package. Also, this will likely lead to everything under the sun getting rebuilt, so it's hard to test fully.