Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable cross compilation for cpython 2.7 #50245

Merged
merged 2 commits into from Nov 15, 2018

Conversation

tathougies
Copy link
Contributor

Motivation for this change

CPython 2.7 cross-compilation currently 'succeeds' but produces a non-working installation. For example, all C modules silently fail to compile. You can see this if you build CPython 2.7 for another architecture (like ARM), and then attempt to import functools in the resulting interpreter. You'll get an error like:

2018-11-10_16:44:02.47232   File "/nix/store/5kfa26ykmihsllf1crz5v56345pv1zkj-python-2.7.15-armv7l-unknown-linux-musleabihf/lib/python2.7/functools.py", line 10, in <module>
2018-11-10_16:44:02.47236     from _functools import partial, reduce
2018-11-10_16:44:02.47244 ImportError: No module named _functools

Python has its own build process, which appears tailored to distributions like ubuntu and debian, not NixOS.

The setup.py script is the culprit here. It attempts to pick up library directories to include on the linker command line. However, it gets confused and picks up the native python library directory. Examining the build output, you'll see that the native modules fail to compile because the libpython2.7.so it finds is not of the right architecture. This patch gets rid of all the setup.py library searching logic and replaces it with the library set nixpkgs has already provided us.

Additionally, the _PYTHON_HOST_PLATFORM env variable apparently needs to be set to get CPython to recognize it's being cross-compiled to begin with.

With these fixes, importing functools now works.

CPython >3 seems to cross-compile fine without any changes.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@@ -176,10 +178,15 @@ let
LIBRARY_PATH = makeLibraryPath paths;
};

# Python 2.7 needs this
crossCompileEnv = if stdenv.hostPlatform != stdenv.buildPlatform
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can simplify this with optionalAttrs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Will do!

@joachifm joachifm added the 6.topic: cross-compilation Building packages on a different sort platform than than they will be run on label Nov 11, 2018
@FRidh FRidh merged commit d0eb502 into NixOS:master Nov 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: cross-compilation Building packages on a different sort platform than than they will be run on 6.topic: python 10.rebuild-darwin: 0 10.rebuild-linux: 1-10
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants