Python: merge expressions of interpreters #53123

FRidh · 2018-12-31T13:20:46Z

Merge CPython expressions
Merge PyPy expressions
Improve cross-compilation of Python packages
Drop python.majorVersion attribute
Introduce derivations for prebuilt PyPy binaries.

Things done

pkgs/top-level/python-packages.nix

pkgs/development/interpreters/python/default.nix

pkgs/top-level/all-packages.nix

FRidh · 2018-12-31T13:45:21Z

cc @LnL7 for Darwin
cc @Mic92 for cross

Mic92 · 2019-01-02T10:08:01Z

I will have a look at cross-compiling once the conflict is resolved.

FRidh · 2019-01-02T12:44:18Z

@Mic92 not sure whether to trust this function:

$(nix-build -A pkgsCross.musl64.python3)/bin/python -c "import platform; print(platform.python_compiler())"
GCC 7.4.0

Mic92 · 2019-01-02T20:13:17Z

@FRidh what's wrong with the output? Afaik we build with gcc for musl.

pkgs/development/interpreters/python/build-python-package-setuptools.nix

FRidh · 2019-01-02T20:21:06Z

yes, I mixed host and build up when I wrote that.

pkgs/development/python-modules/bootstrapped-pip/default.nix

pkgs/development/interpreters/python/build-python-package-setuptools.nix

Ericson2314 · 2019-01-02T20:51:37Z

Very glad this is happening! I started something like the once, but it appears I lost it haha.

Ericson2314

It's looking really good!

Ericson2314 · 2019-01-03T18:27:45Z

pkgs/development/interpreters/python/build-python-package-common.nix

@@ -10,7 +9,7 @@
 , ... } @ attrs:

 attrs // {
-  buildInputs = buildInputs ++ [ bootstrapped-pip ];
+  buildInputs = buildInputs ++ [ python.pythonForBuild.pkgs.bootstrapped-pip ];


Bootstrapped pip is used at build time and so should be a nativeBuildInput.

Alternatively, since you use it explicitly below, maybe it doesn't need to be any sort of imput.

Good catch. I don't think we actually need it as a build input. We call it already from the store path, and Python packages should never call or import from pip. Doesn't mean they won't do it though...

So, we do need it as *buildInput. While it is indeed supposed to be nativeBuildInput, putting it there breaks cross-compilation for me, likely because at the moment something wrong is happening yet it looks alright. I'll keep this issue for another day.

Ericson2314 · 2019-01-03T18:30:41Z

pkgs/development/interpreters/python/build-python-package-common.nix

@@ -24,7 +23,7 @@ attrs // {
    export PYTHONPATH="$out/${python.sitePackages}:$PYTHONPATH"

    pushd dist
-    ${bootstrapped-pip}/bin/pip install *.whl --no-index --prefix=$out --no-cache ${toString installFlags} --build tmpbuild
+    ${python.pythonForBuild.pkgs.bootstrapped-pip}/bin/pip install *.whl --no-index --prefix=$out --no-cache ${toString installFlags} --build tmpbuild


N.B. this is fine, but won't need it once bootstrapped-pip is a nativeBuildInput.

Mic92 · 2019-01-03T20:19:15Z

pkgs/development/interpreters/python/pypy/prebuilt.nix

+    pushd $out
+    ls -l .
+    find {lib,lib_pypy*} -name "*.so" -exec patchelf --replace-needed "libbz2.so.1.0" "libbz2.so.1" {} \;
+    find {lib,lib_pypy*} -name "*.so" -exec patchelf --set-rpath ${stdenv.lib.makeLibraryPath deps} {} \;


In future autoPatchelfHook could make this process easier and less error-prone.

Each time a new major/minor version of CPython was released, a new expression would be written, typically copied from the previous release. Often fixes are only made in the current/latest release. By merging the expressions it's more likely that modifications end up in all versions, as is likely intended. This commit introduces one expression for Python 3, and another for 2.7. These two may also be merged, but it will result in a lot of extra conditionals making the expression harder to follow. A common passthru is introduced for CPython and PyPy. python 2.7: use common passthru

This commit merges the two expressions in a single one, using the passthru function that is shared with CPython.

This changeset allows for cross-compilation of Python packages. Packages built with buildPythonPackage are not allowed to refer to the build machine. Executables that have shebangs will refer to the host.

Drop `python.majorVersion`. For Python language version, use `python.pythonVersion`. For implementation version, use `python.sourceVersion`. Some expressions were broken. Those that were identified were fixed. fixup major

These interpreters are prebuilt by upstream and patched using patchelf. They are primarily added for testing purposes and development on the non-prebuilt PyPy interpreters as it can speed up translation significantly.

FRidh added 6.topic: python 2.status: work-in-progress labels Dec 31, 2018

FRidh force-pushed the singlecpython branch 3 times, most recently from 889b1c8 to 7e13a16 Compare December 31, 2018 13:26

FRidh commented Dec 31, 2018

View reviewed changes

pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved

pkgs/development/interpreters/python/default.nix Outdated Show resolved Hide resolved

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

FRidh force-pushed the singlecpython branch 2 times, most recently from e09dcd7 to baf965e Compare December 31, 2018 13:43

FRidh force-pushed the singlecpython branch 3 times, most recently from 5fa8ca8 to 4a8932b Compare December 31, 2018 17:54

FRidh force-pushed the singlecpython branch from e1b13aa to 7cd7d6e Compare January 2, 2019 12:19

FRidh force-pushed the singlecpython branch from 7cd7d6e to f87d1f1 Compare January 2, 2019 19:10

FRidh mentioned this pull request Jan 2, 2019

WIP: EDK2, OVMF.fd: fixes for cross-compilation + dependencies + armv7l #53064

Closed

FRidh force-pushed the singlecpython branch 2 times, most recently from 92428dc to 9aed1fb Compare January 2, 2019 20:07

Mic92 reviewed Jan 2, 2019

View reviewed changes

pkgs/development/interpreters/python/build-python-package-setuptools.nix Outdated Show resolved Hide resolved

FRidh force-pushed the singlecpython branch from 9aed1fb to 593d9f7 Compare January 2, 2019 20:28

Mic92 reviewed Jan 2, 2019

View reviewed changes

pkgs/development/python-modules/bootstrapped-pip/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 2, 2019

View reviewed changes

pkgs/development/interpreters/python/build-python-package-setuptools.nix Outdated Show resolved Hide resolved

This was referenced Jan 3, 2019

pypy3: remove runtime-wrapper #52635

Closed

Python: fix cross-compilation of extension modules #53320

Closed

FRidh force-pushed the singlecpython branch 2 times, most recently from 999bbb6 to f82cc80 Compare January 3, 2019 16:03

FRidh added 8.has: clean-up 6.topic: cross-compilation Building packages on a different sort platform than than they will be run on 1.severity: mass-rebuild labels Jan 3, 2019

FRidh force-pushed the singlecpython branch 2 times, most recently from e4eca87 to 99faaf7 Compare January 3, 2019 17:50

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Jan 3, 2019

FRidh changed the title ~~WIP: CPython: merge expressions of interpreters into single expression~~ Python: merge expressions of interpreters Jan 3, 2019

Ericson2314 reviewed Jan 3, 2019

View reviewed changes

FRidh mentioned this pull request Jan 3, 2019

systemd fails to cross-compile #53207

Closed

Mic92 reviewed Jan 3, 2019

View reviewed changes

FRidh force-pushed the singlecpython branch from 99faaf7 to ff202fe Compare January 4, 2019 09:36

FRidh added 5 commits January 4, 2019 10:45

pypy: merge 2.7 and 3.5 into a single expression

613498a

This commit merges the two expressions in a single one, using the passthru function that is shared with CPython.

Python: improve cross-compilation

f665828

This changeset allows for cross-compilation of Python packages. Packages built with buildPythonPackage are not allowed to refer to the build machine. Executables that have shebangs will refer to the host.

Python: drop python.majorVersion

0a2caa4

Drop `python.majorVersion`. For Python language version, use `python.pythonVersion`. For implementation version, use `python.sourceVersion`. Some expressions were broken. Those that were identified were fixed. fixup major

pythonInterpreters.pypy{27,35}_prebuilt: init at 6.0.0

d91b496

These interpreters are prebuilt by upstream and patched using patchelf. They are primarily added for testing purposes and development on the non-prebuilt PyPy interpreters as it can speed up translation significantly.

FRidh force-pushed the singlecpython branch from ff202fe to d91b496 Compare January 4, 2019 09:45

FRidh merged commit 29ee864 into NixOS:staging Jan 4, 2019

FRidh deleted the singlecpython branch January 4, 2019 11:23

andersk mentioned this pull request Jan 15, 2019

pypy: Correct Python version in meta description #54015

Merged

10 tasks

bb010g mentioned this pull request Apr 17, 2019

python3{5,6}: Reintroduce bytecode determinism #59796

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: merge expressions of interpreters #53123

Python: merge expressions of interpreters #53123

FRidh commented Dec 31, 2018 •

edited

FRidh commented Dec 31, 2018

Mic92 commented Jan 2, 2019

FRidh commented Jan 2, 2019

Mic92 commented Jan 2, 2019

FRidh commented Jan 2, 2019

Ericson2314 commented Jan 2, 2019

Ericson2314 left a comment

Ericson2314 Jan 3, 2019

Ericson2314 Jan 3, 2019

FRidh Jan 3, 2019

FRidh Jan 4, 2019

Ericson2314 Jan 3, 2019

Mic92 Jan 3, 2019

Navigation Menu

Python: merge expressions of interpreters #53123

Python: merge expressions of interpreters #53123

Conversation

FRidh commented Dec 31, 2018 • edited

Things done

FRidh commented Dec 31, 2018

Mic92 commented Jan 2, 2019

FRidh commented Jan 2, 2019

Mic92 commented Jan 2, 2019

FRidh commented Jan 2, 2019

Ericson2314 commented Jan 2, 2019

Ericson2314 left a comment

Choose a reason for hiding this comment

Ericson2314 Jan 3, 2019

Choose a reason for hiding this comment

Ericson2314 Jan 3, 2019

Choose a reason for hiding this comment

FRidh Jan 3, 2019

Choose a reason for hiding this comment

FRidh Jan 4, 2019

Choose a reason for hiding this comment

Ericson2314 Jan 3, 2019

Choose a reason for hiding this comment

Mic92 Jan 3, 2019

Choose a reason for hiding this comment

FRidh commented Dec 31, 2018 •

edited