cpython: fix cross-compiling extension modules #98915

lopsided98 · 2020-09-27T05:03:21Z

Motivation for this change

Fixes cross-compiling Python native extension modules to host platforms with a different word length than the build platform as well as in a few other cases.

Currently, cross-compiling Python extension modules generally works assuming the build and host platforms have the same word size. This leads to problems like those seen here: #50925 (comment), where x86_64 -> aarch64 works, but x86_64 -> armv7l doesn't. Unfortunately, this PR doesn't fix btrfs-progs (#50925) and talloc (#63043), because they do not use setuptools/distutils.

This happens because the build Python's headers are used, rather than host Python's headers. The header location and other data used for compiling extension modules are available in the _sysconfigdata__<kernel>_<platform> module. distutils.sysconfig can be patched to use this data and then Python needs to be told to use sysconfigdata from host Python using the _PYTHON_SYSCONFIGDATA_NAME environment variable. I believe the patch should be upstreamable.

There was an ill-fated upstream PR to improve and document Python cross-compilation, avoiding the need to manually construct _PYTHON_SYSCONFIGDATA_NAME, but based on my quick review I don't think it would have worked for us as is: python/cpython#17420

Setting _PYTHON_HOST_PLATFORM and _PYTHON_SYSCONFIGDATA_NAME appropriately also fixes some other issues, such as adding the correct platform suffix to extension module file names. This allows cryptography to build after a few minor fixes.

I have tested this PR by cross-compiling a number of Python packages for armv6l, including requests and netifaces, both of which failed before.

Things done

lopsided98 · 2020-09-27T05:05:00Z

cc @FRidh @Ericson2314

pkgs/development/interpreters/python/mk-python-derivation.nix

lopsided98 · 2020-09-27T05:11:05Z

pkgs/development/python-modules/cryptography/default.nix

@@ -31,6 +32,7 @@ buildPythonPackage rec {

  outputs = [ "out" "dev" ];

+  nativeBuildInputs = stdenv.lib.optional (!isPyPy) python.pythonForBuild.pkgs.cffi;


Splicing doesn't seem to work here. Anyone know why?

If this is always the case we are going to have a bit of a problem, because all the nativeBuildInputs in case of Python packages are then incorrect.

pkgs/development/interpreters/python/mk-python-derivation.nix

FRidh · 2020-09-27T08:29:40Z

pkgs/development/python-modules/cryptography/default.nix

@@ -31,6 +32,7 @@ buildPythonPackage rec {

  outputs = [ "out" "dev" ];

+  nativeBuildInputs = stdenv.lib.optional (!isPyPy) python.pythonForBuild.pkgs.cffi;


lopsided98 · 2020-09-28T20:51:49Z

I created an upstream PR: python/cpython#22440

lopsided98 · 2020-10-01T05:24:48Z

I have now implemented this PR as a setup hook. I did not add it to the common Python setup-hook.sh because the environment variables are cpython specific and because I needed access to cpython's $out, which cannot be done easily if the setup hook is defined in a separate derivation. To get the sysconfigdata module on PYTHONPATH, I moved it to site-packages.

Also, this PR now fixes #50925.

lopsided98 · 2020-10-01T19:43:52Z

After looking into it some more, this PR also helps solve #63043. Normally talloc uses python-config, but it will fall back to using the sysconfig module if python-config is not found (apparently this is to support Windows, but it seems to work fine on Linux). You can force this to happen by setting PYTHON_CONFIG=/invalid in the environment. This also applies to samba and tdb, which all use the same build system.

Thra11 · 2020-10-02T13:10:30Z

This doesn't fix #63043 for me. Running nix-build '<nixpkgs>' -A pkgsCross.armv7l-hf-multiplatform.talloc I still get:

Checking if compiler accepts -fstack-clash-protection                             : yes
Checking linker accepts -Wl,-no-undefined                                         : yes
Checking linker accepts ['-undefined', 'dynamic_lookup']                          : no
Checking for program 'python3'                                                    : /nix/store/0w5g6pmsb0i7f7vklbg2snyj3wfalq7y-python3-3.8.5/bin/python3
Checking for program 'python'                                                     : /nix/store/0w5g6pmsb0i7f7vklbg2snyj3wfalq7y-python3-3.8.5/bin/python3
Checking for program 'python3'                                                    : /nix/store/0w5g6pmsb0i7f7vklbg2snyj3wfalq7y-python3-3.8.5/bin/python3
Checking for python version >= 3.5.0                                              : 3.8.5
python-config                                                                     : /nix/store/0w5g6pmsb0i7f7vklbg2snyj3wfalq7y-python3-3.8.5/bin/python3-config
Asking python-config for pyembed '--cflags --libs --ldflags --embed' flags        : yes
Testing pyembed configuration                                                     : Could not build a python embedded interpreter
Testing pyembed configuration                                                     : Could not build a python embedded interpreter
The configuration failed
(complete log in /build/talloc-2.3.1/bin/config.log)
builder for '/nix/store/5bn9ss7a1al5js5l0ncz0n4q5i3hqgc9-talloc-2.3.1-armv7l-unknown-linux-gnueabihf.drv' failed with exit code 1

Thra11 · 2020-10-02T13:11:19Z

Unfortunately, --keep-failed doesn't seem to work any more, so I can't check config.log.

lopsided98 · 2020-10-02T16:54:12Z

Yes, it doesn't solve #63043 by itself, you also need to set PYTHON_CONFIG="/invalid" in talloc.

Thra11 · 2020-10-02T17:07:11Z

Ah. Missed that bit, sorry!

Thra11 · 2020-10-02T17:20:38Z

Still doesn't quite work for me:

Checking for python version >= 3.5.0                                              : 3.8.5
python-config                                                                     : /invalid
Checking for library python3.8 in LIBPATH_PYEMBED                                 : not found
Checking for library python3.8 in LIBDIR                                          : not found
Checking for library python3.8 in python_LIBPL                                    : yes
Checking for header Python.h                                                      : Distutils not installed? Broken python installation? Get python-config now!
The configuration failed
(complete log in /build/talloc-2.3.1/bin/config.log)
builder for '/nix/store/8jf92la0x7x00wfaffka361s1qipggcb-talloc-2.3.1-armv7l-unknown-linux-gnueabihf.drv' failed with exit code 1

lopsided98 · 2020-10-02T17:25:55Z

Oh right, you also have to add both host and build Python as dependencies. I've added the full fix to this PR.

lopsided98 · 2020-11-29T05:59:41Z

I fixed the older Python versions as well now.

@ofborg build python36.tests python37.tests python38.tests python39.tests python310.tests

lopsided98 · 2020-11-29T18:25:38Z

3.10 was already broken.

@ofborg build python36.tests python37.tests python38.tests python39.tests

FRidh · 2020-11-29T18:48:47Z

Currently we're working on updating the Python packages set. When that is done, I'll push this job to that branch for further testing. That will be about a week from now.

Atemu · 2020-12-14T04:31:10Z

Hi, it's been two weeks; any update?

flokli · 2020-12-26T17:30:30Z

I can confirm this fixes #107578.

I cherry-picked this on top of currently master locally and will run some more tests.

@FRidh is this pushed somewhere to a jobset?

flokli · 2020-12-26T19:20:33Z

I pushed the cherry-pick to cpython-headers-cross.

~~This reproducibly breaks the test of libuv (on x86_64-linux), which is a dependency of cmake:~~

not ok 72 - fs_fstat
# exit code 134
# Output from process `fs_fstat`:
# Assertion failed in test/test-fs.c on line 1389: s->st_dev == (uint64_t) t.st_dev

pkgs.pkgsCross.raspberryPi.libuv builds, however.

Edit: Ouch, ignore that. Apparently, this seems to be have been a bitflip.

flokli · 2020-12-27T13:38:51Z

I was able to successfully build -A python36.tests -A python37.tests -A python38.tests -A python39.tests.

python310.tests failed before and after this.

FRidh · 2020-12-27T14:02:14Z

Pushed 0959552 to python-unstable. Should make it easier to test wider. https://hydra.nixos.org/eval/1637790

FRidh · 2020-12-28T07:52:41Z

https://hydra.nixos.org/eval/1637790 looks good. This now needs a rebase and then it can go in staging.

flokli · 2020-12-28T12:56:59Z

I rebased this to latest staging and pushed again.

The only conflict was the addition of the # Backport a fix for ctypes.util.find_library. patch in pkgs/development/interpreters/python/cpython/default.nix in staging, which was causing git to not find the exact place to add the patches introduced here.

flokli · 2020-12-28T13:22:15Z

I successfully built all of python3{6,7,8,9}.tests and confirmed this still fixes pkgsCross.raspberryPi.libselinux, so it doesn't look like there's any conflicts due to other changes in staging in the meantime. Let's get this in :-)

matthewbauer · 2020-12-29T00:53:16Z

Would it be possible to apply this to 20.09? I think we may be able to avoid the mass-rebuild...

jonringer · 2020-12-29T00:59:48Z

as long it doesn't have any breakages. I'm fine with back-porting it

Ericson2314 · 2020-12-29T02:59:20Z

@matthewbauer While the first version of this was made independently, it's possible the final version relies on #104201 in some ways, too.

FRidh · 2020-12-29T07:39:55Z

A backport would need extensive testing as well first.

…

On Tue, 29 Dec 2020, 01:53 Matthew Bauer, ***@***.***> wrote: Would it be possible to apply this to 20.09? I think we may be able to avoid the mass-rebuild... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#98915 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQHZ34IC74C5UOK37MP4GDSXESATANCNFSM4R3KRSSA> .

jonringer · 2020-12-29T16:05:01Z

A backport would need extensive testing as well first.

This is my biggest concern. I would prefer not to backport if the scope of everything needed to backport is ill-defined, and likely to introduce regressions. Unstable works great for iteration, but the purpose of the release channel is to allow people to update without fear.

pjjw · 2021-01-21T09:09:49Z

hey just FYI i think this PR broke python extensions under musl libc, have a PR out to fix at #110293. looks like you all are talking about backporting this and it'd be a bummer to backport that breakage too.

lopsided98 requested review from FRidh and jonringer as code owners September 27, 2020 05:03

ofborg bot added the 6.topic: python label Sep 27, 2020

lopsided98 commented Sep 27, 2020

View reviewed changes

pkgs/development/interpreters/python/mk-python-derivation.nix Outdated Show resolved Hide resolved

lopsided98 commented Sep 27, 2020

View reviewed changes

pkgs/development/interpreters/python/mk-python-derivation.nix Outdated Show resolved Hide resolved

lopsided98 commented Sep 27, 2020

View reviewed changes

ofborg bot added 10.rebuild-darwin-stdenv 10.rebuild-linux-stdenv labels Sep 27, 2020

ofborg bot requested a review from primeos September 27, 2020 06:56

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Sep 27, 2020

FRidh requested changes Sep 27, 2020

View reviewed changes

bjornfor added the 6.topic: cross-compilation Building packages on a different sort platform than than they will be run on label Sep 27, 2020

lopsided98 force-pushed the python-cross-extensions branch from 2f56545 to 4571e76 Compare September 29, 2020 17:05

ofborg bot requested a review from FRidh September 29, 2020 19:20

lopsided98 mentioned this pull request Sep 30, 2020

pythonPackages.protobuf: fix cross-compilation #99141

Merged

10 tasks

lopsided98 force-pushed the python-cross-extensions branch from 4571e76 to 82c7ed5 Compare October 1, 2020 05:01

lopsided98 mentioned this pull request Oct 1, 2020

pkgsCross.armv7l-hf-multiplatform.btrfs-progs fails to build #50925

Closed

FRidh added this to To do in Python batch upgrade Dec 2, 2020

Atemu mentioned this pull request Dec 12, 2020

The NixOS SD image cannot be cross-compiled #106759

Closed

9 tasks

Atemu mentioned this pull request Dec 25, 2020

libselinux doesn't cross-compile for armv6/7 #107578

Closed

flokli mentioned this pull request Dec 26, 2020

libuv fails to build on certain systems #107657

Closed

cpython: fix finding headers when cross-compiling extension modules

1a65c5d

flokli force-pushed the python-cross-extensions branch from cf2bb6e to 1a65c5d Compare December 28, 2020 12:56

flokli merged commit 093632a into NixOS:staging Dec 28, 2020

Python batch upgrade automation moved this from To do to Done Dec 28, 2020

FRidh mentioned this pull request Jan 7, 2021

Staging next #108423

Merged

10 tasks

pjjw mentioned this pull request Jan 21, 2021

cpython: fix extensions when using a musl toolchain #110293

Merged

10 tasks

lopsided98 deleted the python-cross-extensions branch January 21, 2021 23:53

lopsided98 mentioned this pull request Feb 15, 2021

Fix finding headers when cross compiling pypa/setuptools#2569

Closed

2 tasks

lopsided98 mentioned this pull request Mar 14, 2021

Fix finding headers when cross compiling pypa/distutils#35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cpython: fix cross-compiling extension modules #98915

cpython: fix cross-compiling extension modules #98915

lopsided98 commented Sep 27, 2020

lopsided98 commented Sep 27, 2020

lopsided98 Sep 27, 2020

FRidh Sep 27, 2020

FRidh Oct 6, 2020

FRidh Sep 27, 2020

lopsided98 commented Sep 28, 2020

lopsided98 commented Oct 1, 2020

lopsided98 commented Oct 1, 2020

Thra11 commented Oct 2, 2020

Thra11 commented Oct 2, 2020

lopsided98 commented Oct 2, 2020 •

edited

Thra11 commented Oct 2, 2020

Thra11 commented Oct 2, 2020

lopsided98 commented Oct 2, 2020

lopsided98 commented Nov 29, 2020

lopsided98 commented Nov 29, 2020 •

edited

FRidh commented Nov 29, 2020

Atemu commented Dec 14, 2020

flokli commented Dec 26, 2020

flokli commented Dec 26, 2020 •

edited

flokli commented Dec 27, 2020

FRidh commented Dec 27, 2020 •

edited

FRidh commented Dec 28, 2020

flokli commented Dec 28, 2020

flokli commented Dec 28, 2020

matthewbauer commented Dec 29, 2020

jonringer commented Dec 29, 2020 •

edited

Ericson2314 commented Dec 29, 2020

FRidh commented Dec 29, 2020 via email

jonringer commented Dec 29, 2020

pjjw commented Jan 21, 2021

		@@ -31,6 +32,7 @@ buildPythonPackage rec {

		outputs = [ "out" "dev" ];

		nativeBuildInputs = stdenv.lib.optional (!isPyPy) python.pythonForBuild.pkgs.cffi;

cpython: fix cross-compiling extension modules #98915

cpython: fix cross-compiling extension modules #98915

Conversation

lopsided98 commented Sep 27, 2020

Motivation for this change

Things done

lopsided98 commented Sep 27, 2020

lopsided98 Sep 27, 2020

Choose a reason for hiding this comment

FRidh Sep 27, 2020

Choose a reason for hiding this comment

FRidh Oct 6, 2020

Choose a reason for hiding this comment

FRidh Sep 27, 2020

Choose a reason for hiding this comment

lopsided98 commented Sep 28, 2020

lopsided98 commented Oct 1, 2020

lopsided98 commented Oct 1, 2020

Thra11 commented Oct 2, 2020

Thra11 commented Oct 2, 2020

lopsided98 commented Oct 2, 2020 • edited

Thra11 commented Oct 2, 2020

Thra11 commented Oct 2, 2020

lopsided98 commented Oct 2, 2020

lopsided98 commented Nov 29, 2020

lopsided98 commented Nov 29, 2020 • edited

FRidh commented Nov 29, 2020

Atemu commented Dec 14, 2020

flokli commented Dec 26, 2020

flokli commented Dec 26, 2020 • edited

flokli commented Dec 27, 2020

FRidh commented Dec 27, 2020 • edited

FRidh commented Dec 28, 2020

flokli commented Dec 28, 2020

flokli commented Dec 28, 2020

matthewbauer commented Dec 29, 2020

jonringer commented Dec 29, 2020 • edited

Ericson2314 commented Dec 29, 2020

FRidh commented Dec 29, 2020 via email

jonringer commented Dec 29, 2020

pjjw commented Jan 21, 2021

lopsided98 commented Oct 2, 2020 •

edited

lopsided98 commented Nov 29, 2020 •

edited

flokli commented Dec 26, 2020 •

edited

FRidh commented Dec 27, 2020 •

edited

jonringer commented Dec 29, 2020 •

edited