Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

buildPythonPackage: recompile bytecode for reproducibility #90208

Merged
merged 2 commits into from
Jun 13, 2020
Merged

buildPythonPackage: recompile bytecode for reproducibility #90208

merged 2 commits into from
Jun 13, 2020
+42 −4

Conversation

FRidh
Copy link
Member

@FRidh FRidh commented Jun 12, 2020

Due to a change in pip the unpacked wheels are no longer reproducible.
We recompile the bytecode to cleanup this error.
#81441

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Sorry, something went wrong.

@FRidh FRidh requested a review from jonringer as a code owner June 12, 2020 19:44
@FRidh FRidh mentioned this pull request Jun 12, 2020
Closed
@jonringer
Copy link
Contributor

@GrahamcOfBorg build python3Packages.requests
@GrahamcOfBorg build azure-cli

@misuzu
Copy link
Contributor

misuzu commented Jun 12, 2020

pycrypto is not building:

$ nix-build . -A pkgs.python37Packages.pycrypto
these derivations will be built:
  /nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv
building '/nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv'...
Sourcing python-recompile-bytecode-hook.sh
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
creating pycrypto.egg-info
writing pycrypto.egg-info/PKG-INFO
writing dependency_links to pycrypto.egg-info/dependency_links.txt
writing requirements to pycrypto.egg-info/requires.txt
writing top-level names to pycrypto.egg-info/top_level.txt
writing manifest file 'pycrypto.egg-info/SOURCES.txt'
reading manifest file 'pycrypto.egg-info/SOURCES.txt'
writing manifest file 'pycrypto.egg-info/SOURCES.txt'
Copying pycrypto.egg-info to build/bdist.linux-x86_64/wheel/pycrypto-3.9.7-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/pycrypto-3.9.7.dist-info/WHEEL
creating 'dist/pycrypto-3.9.7-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'pycrypto-3.9.7.dist-info/METADATA'
adding 'pycrypto-3.9.7.dist-info/WHEEL'
adding 'pycrypto-3.9.7.dist-info/top_level.txt'
adding 'pycrypto-3.9.7.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Finished executing setuptoolsBuildPhase
installing
Executing pipInstallPhase
/build/dist /build
Processing ./pycrypto-3.9.7-py3-none-any.whl
Requirement already satisfied: pycryptodome in /nix/store/6qjnk13bsxhcqgbqw1qsz1y0l8cxh5bv-python3.7-pycryptodome-3.9.7/lib/python3.7/site-packages (from pycrypto==3.9.7) (3.9.7)
Installing collected packages: pycrypto
Successfully installed pycrypto-3.9.7
/build
Finished executing pipInstallPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7
strip is /nix/store/bh3r88sv8wckwmfyhjxbqmxcha0hrm8h-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7/lib
patching script interpreter paths in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7
checking for references to /build/ in /nix/store/gjch87j13jcx1kbazjsl8rxzvywr8fwf-python3.7-pycrypto-3.9.7...
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
pythonRecompileBytecodePhase
*** Error compiling '/nix/store/6xmiwapzvgx4a444b9qaz5ldfv2jqw1c-python3-3.7.7/lib/python3.7/site-packages/sitecustomize.py'...
PermissionError: [Errno 13] Permission denied: '/nix/store/6xmiwapzvgx4a444b9qaz5ldfv2jqw1c-python3-3.7.7/lib/python3.7/site-packages/__pycache__/sitecustomize.cpython-37.opt-2.pyc.140737341708784'
builder for '/nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv' failed with exit code 123
error: build of '/nix/store/rb8djlzgnm3sb3l56z8976rnw8w3x8yq-python3.7-pycrypto-3.9.7.drv' failed

@jonringer
Copy link
Contributor

maybe exclude the top level of site-packages?

@jonringer
Copy link
Contributor

actually, figured out root cause, some packages don't ship any .py's, and in that case, it errors out

@FRidh
Copy link
Member Author

FRidh commented Jun 13, 2020

Ah yes, pycrypto, my dummy package providing pycryptodome. Would be nice if we could finally get rid of it.

@FRidh
Copy link
Member Author

FRidh commented Jun 13, 2020

Thanks for the feedback, it's building now.

@FRidh
Copy link
Member Author

FRidh commented Jun 13, 2020

Oh, now we have another failure

$ nix-build -A python2.pkgs.setuptools
...
option -j not recognized
usage: python compileall.py [-l] [-f] [-q] [-d destdir] [-x regexp] [-i list] [directory|file ...]

arguments: zero or more file and directory names to compile; if no arguments given,
           defaults to the equivalent of -l sys.path

options:
-l: don't recurse into subdirectories
-f: force rebuild even if timestamps are up-to-date
-q: output only error messages
-d destdir: directory to prepend to file paths for use in compile-time tracebacks and in
            runtime tracebacks in cases where the source file is unavailable
-x regexp: skip files matching the regular expression regexp; the regexp is searched for
           in the full path of each file considered for compilation
-i file: add all the files and directories listed in file to the list considered for
         compilation; if "-", names are read from stdin
builder for '/nix/store/9hrz8wjl5ij1psq4b47yj0gbc4w3p3wm-python2.7-setuptools-44.0.0.drv' failed with exit code 1

Seems like Python 2 version does not have -j

@FRidh
buildPythonPackage: recompile bytecode for reproducibility
818cf78 
Due to a change in pip the unpacked wheels are no longer reproducible.
We recompile the bytecode to cleanup this error.
NixOS#81441
@FRidh FRidh merged commit da12111 into NixOS:staging Jun 13, 2020
@FRidh FRidh deleted the bytecode branch June 15, 2020 10:40
@FRidh
Copy link
Member Author

FRidh commented Jun 15, 2020

gtk-doc fails with

pythonRecompileBytecodePhase
find: ‘/nix/store/i5cla9rwy1kaakwy7k4vdjgrlw6qawrj-gtk-doc-1.32/lib/python3.8/site-packages’: No such file or directory
builder for '/nix/store/7clc8qpva22qcq06w0fw9hiccicfziiv-gtk-doc-1.32.drv' failed with exit code 1

https://hydra.nixos.org/build/122153227

For now I'll pass

FRidh added a commit that referenced this pull request Jun 15, 2020
@FRidh
gtk-doc: dont recompile bytecode
4ece345 
Would not build otherwise. The hooks needs to be fixed.
#90208 (comment)
@FRidh FRidh mentioned this pull request Jun 15, 2020
Closed
@FRidh
Copy link
Member Author

FRidh commented Jun 15, 2020

Issue tracking regressions #90516.

@FRidh
Copy link
Member Author

FRidh commented Jun 15, 2020

@adisbladis @DavHau you may want to pass dontUsePythonRecompileBytecode = true; in poetry2nix and mach-nix. I don't expect trouble with Python 3 but there are already some Python 2 regressions. Depending on how many there are it may be disabled altogether for Python 2.

@misuzu misuzu mentioned this pull request Jun 22, 2020
Merged
17 tasks
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-builds-keep-pyc-in-store-dockertools/8784/6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonringer jonringer

Assignees
No one assigned
Labels
6.topic: python 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@FRidh @jonringer @misuzu @nixos-discourse