Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3: add C++ compiler support for distutils #39576

Merged
merged 1 commit into from May 9, 2018
Merged

python3: add C++ compiler support for distutils #39576

merged 1 commit into from May 9, 2018

Conversation

veprbl
Copy link
Member

@veprbl veprbl commented Apr 26, 2018

This implements 095095c ('python: add C++ compiler support for distutils')
for python3. Should fix various problems with python packages on darwin.

Motivation for this change

Even in python3 the distutils will try to invoke C compiler to compile C++ files, which works against GCC and Apple Clang, but not with clang that we ship.

This expands on #27649 to play well with numpy/cython (see 46dd9df)

Fixes: #27649

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@veprbl veprbl requested a review from FRidh as a code owner April 26, 2018 22:05
@veprbl veprbl mentioned this pull request Apr 26, 2018
Closed
8 tasks
@veprbl
python3: add C++ compiler support for distutils
31010e0 
This implements 095095c ('python: add C++ compiler support for distutils')
for python3. Should fix various problems with python packages on darwin.
@7c6f434c 7c6f434c mentioned this pull request Apr 26, 2018
Closed
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python3

Partial log (click to expand)

strip is /nix/store/jfnxjmhyjin5h5mz8w26g9mq0x8gy5kg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib  /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/bin
patching script interpreter paths in /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/bin/python3.6m-config: interpreter directive changed from "/bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib/python3.6/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib/python3.6/config-3.6m-x86_64-linux-gnu/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib/python3.6/config-3.6m-x86_64-linux-gnu/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5...
wrong ELF type
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python3

Partial log (click to expand)

strip is /nix/store/i3k26ygq416sr5ng9f87041f3zcid9di-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib  /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/bin
patching script interpreter paths in /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib/python3.6/config-3.6m-aarch64-linux-gnu/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib/python3.6/config-3.6m-aarch64-linux-gnu/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib/python3.6/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/bin/python3.6m-config: interpreter directive changed from "/bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5...
wrong ELF type
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python3

Partial log (click to expand)

/nix/store/m00ppbhglkn98qiwafbqgh7fif0w9six-python3-3.6.5

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python3

Partial log (click to expand)

/nix/store/gpqyw2sm11kmzrdnpw7pgdr0y7h1813x-python3-3.6.5

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python3

Partial log (click to expand)

strip is /nix/store/bbm9qyr0mbpy46nvsz27kb06v8z19bx0-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/8axs3yl256g4wpibm6p7iypv0r7cn4pg-curl-7.59.0-devdoc
building '/nix/store/d62swpidd2ipkyq9pqvmyhhn5rq79brb-python-3.x-distutils-C++.patch.drv'...

trying https://bugs.python.org/file47046/python-3.x-distutils-C++.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10947  100 10947    0     0  26635      0 --:--:-- --:--:-- --:--:-- 26700
cannot build derivation '/nix/store/rr5bm03pza8qfn58bl14g382miwccbfy-python3-3.6.5.drv': 2 dependencies couldn't be built
error: build of '/nix/store/rr5bm03pza8qfn58bl14g382miwccbfy-python3-3.6.5.drv' failed

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python3

Partial log (click to expand)

/nix/store/7jsv3azj6gdxwscbq4h81b5s9lafvanc-Security-osx-10.9.5/Library/Frameworks/Security.framework/Headers/cssmtype.h:2023:3: note: 'CSSM_QUERY' has been explicitly marked deprecated here
} CSSM_QUERY DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER, *CSSM_QUERY_PTR DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER;
  ^
In file included from DeviceOnHold.c:38:
In file included from ./SystemConfiguration.framework/Headers/SystemConfiguration.h:121:
In file included from ./SystemConfiguration.framework/Headers/SCPreferences.h:38:
In file included from /nix/store/7jsv3azj6gdxwscbq4h81b5s9lafvanc-Security-osx-10.9.5/Library/Frameworks/Security.framework/Headers/Security.h:42:
building of '/nix/store/f0kppx79iwfxvvpjm133bxnv3fbrilaa-configd-osx-10.8.5.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/q19y11kzv19fwfn30zdsrs1l2rwqs98k-python3-3.6.5.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/q19y11kzv19fwfn30zdsrs1l2rwqs98k-python3-3.6.5.drv' failed

@veprbl
Copy link
Member Author

veprbl commented Apr 27, 2018

I've verified that hasDistutilsCxxPatch is needed for tests of python3Packages.cython to succeed (same as I was previously seeing for python2Packages.cython). Perhaps this was not caught in #27649 since the tests were completely disabled until recently (#37396).

@knedlsepp knedlsepp mentioned this pull request Apr 29, 2018
Closed
8 tasks
@knedlsepp
Copy link
Member

@FRidh: This would unbreak Cython on darwin.

knedlsepp
knedlsepp reviewed Apr 29, 2018
View reviewed changes
pkgs/development/interpreters/python/cpython/3.4/default.nix
@@ -39,6 +39,8 @@ let
++ optionals x11Support [ tcl tk libX11 xproto ]
++ optionals stdenv.isDarwin [ CF configd ];

hasDistutilsCxxPatch = !(stdenv.cc.isGNU or false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about needsDistutilsCxxPatch?

Copy link
Member Author

@veprbl veprbl Apr 29, 2018 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@knedlsepp The meaning depends on the context. The point to have this attribute in the first place is to query python.hasDistutilsCxxPatch from numpy so that its numpy counterpart can be engaged (it won't work otherwise). But PR's are welcome if you still want to change it.

@knedlsepp knedlsepp mentioned this pull request Apr 29, 2018
Closed
@knedlsepp
Copy link
Member

@FRidh Any concerns?

@matthewbauer matthewbauer merged commit 1d3fe7d into NixOS:staging May 9, 2018
@vcunat
Copy link
Member

vcunat commented Jul 3, 2018

After minor update 78e0521 the patch no longer applies (staging).

@vcunat vcunat mentioned this pull request Jul 3, 2018
Merged
8 tasks
@veprbl
Copy link
Member Author

veprbl commented Jul 3, 2018

I guess this is caused by python/cpython@10f715d

@veprbl veprbl mentioned this pull request Jul 3, 2018
Merged
9 tasks
@veprbl veprbl deleted the pr/py3_distutils_fix branch December 1, 2020 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@knedlsepp knedlsepp knedlsepp left review comments

@FRidh FRidh Awaiting requested review from FRidh

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 501+ 10.rebuild-linux: 0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@veprbl @GrahamcOfBorg @knedlsepp @vcunat @matthewbauer