Fix clblas and libgpuarray on darwin #38971

knedlsepp · 2018-04-15T17:35:35Z

Motivation for this change

This enables the builds for clblas and libgpuarray, which consequently should also lead to
working python2Packages.Theano/python3Packages.Theano.
This fixes the issue #33073.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Mic92 · 2018-04-15T18:29:37Z

@GrahamcOfBorg build python3Packages.Theano python2Packages.Theano python27Packages.libgpuarray python3Packages.libgpuarray clblas

GrahamcOfBorg · 2018-04-15T19:29:47Z

Failure on aarch64-linux (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

  ^~~~~~~
acquired default
acquired Global_A
building of '/nix/store/g9ccyb85m5kxbfb3gy26840d90qxsf9k-python2.7-Cython-0.28.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/qvqv66l4irjx0q8is7r5s8gqas31wx44-python2.7-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vvq3v4s85dyrv8qwz7ajw2d9inaqcwvq-python2.7-Theano-1.0.1.drv': 1 dependencies couldn't be built
building of '/nix/store/l60jan1mgs2yd0lnkmsprzjapx4rynv9-python3.6-Cython-0.28.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/sa9kks3cph0dl1ak04azzzp9ln0m9hsv-python3.6-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/grwj7zyg7nwvwwgglgr6w4jgcrpfk9ac-python3.6-Theano-1.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/grwj7zyg7nwvwwgglgr6w4jgcrpfk9ac-python3.6-Theano-1.0.1.drv', '/nix/store/qvqv66l4irjx0q8is7r5s8gqas31wx44-python2.7-libgpuarray-0.7.5.drv', '/nix/store/sa9kks3cph0dl1ak04azzzp9ln0m9hsv-python3.6-libgpuarray-0.7.5.drv', '/nix/store/vvq3v4s85dyrv8qwz7ajw2d9inaqcwvq-python2.7-Theano-1.0.1.drv' failed

GrahamcOfBorg · 2018-04-15T20:29:51Z

Failure on aarch64-linux (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

  ^~~~~~~
acquired default
acquired Global_A
building of '/nix/store/g9ccyb85m5kxbfb3gy26840d90qxsf9k-python2.7-Cython-0.28.1.drv' timed out after 3600 seconds
building of '/nix/store/l60jan1mgs2yd0lnkmsprzjapx4rynv9-python3.6-Cython-0.28.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/qvqv66l4irjx0q8is7r5s8gqas31wx44-python2.7-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/sa9kks3cph0dl1ak04azzzp9ln0m9hsv-python3.6-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vvq3v4s85dyrv8qwz7ajw2d9inaqcwvq-python2.7-Theano-1.0.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/grwj7zyg7nwvwwgglgr6w4jgcrpfk9ac-python3.6-Theano-1.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/grwj7zyg7nwvwwgglgr6w4jgcrpfk9ac-python3.6-Theano-1.0.1.drv', '/nix/store/qvqv66l4irjx0q8is7r5s8gqas31wx44-python2.7-libgpuarray-0.7.5.drv', '/nix/store/sa9kks3cph0dl1ak04azzzp9ln0m9hsv-python3.6-libgpuarray-0.7.5.drv', '/nix/store/vvq3v4s85dyrv8qwz7ajw2d9inaqcwvq-python2.7-Theano-1.0.1.drv' failed

GrahamcOfBorg · 2018-04-15T21:29:56Z

Failure on aarch64-linux (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

memoryview_inplace_division.cpp:1599:28: note: in expansion of macro ‘PyMem_NEW’
         op->data.ob_item = PyMem_NEW(char, nbytes);
                            ^~~~~~~~~
building of '/nix/store/g9ccyb85m5kxbfb3gy26840d90qxsf9k-python2.7-Cython-0.28.1.drv' timed out after 3600 seconds
building of '/nix/store/l60jan1mgs2yd0lnkmsprzjapx4rynv9-python3.6-Cython-0.28.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/qvqv66l4irjx0q8is7r5s8gqas31wx44-python2.7-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/sa9kks3cph0dl1ak04azzzp9ln0m9hsv-python3.6-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vvq3v4s85dyrv8qwz7ajw2d9inaqcwvq-python2.7-Theano-1.0.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/grwj7zyg7nwvwwgglgr6w4jgcrpfk9ac-python3.6-Theano-1.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/grwj7zyg7nwvwwgglgr6w4jgcrpfk9ac-python3.6-Theano-1.0.1.drv', '/nix/store/qvqv66l4irjx0q8is7r5s8gqas31wx44-python2.7-libgpuarray-0.7.5.drv', '/nix/store/sa9kks3cph0dl1ak04azzzp9ln0m9hsv-python3.6-libgpuarray-0.7.5.drv', '/nix/store/vvq3v4s85dyrv8qwz7ajw2d9inaqcwvq-python2.7-Theano-1.0.1.drv' failed

GrahamcOfBorg · 2018-04-15T22:57:48Z

Failure on x86_64-darwin (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
pep448_test_extcall.cpp:12571:26: warning: code will never be executed [-Wunreachable-code]
                module = PyImport_ImportModuleLevelObject(
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
building of '/nix/store/x9ff903kpnmmvf39cdw4pbp08rl30mmh-python3.6-Cython-0.28.1.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/33fsl0n5nmyiwvkzq4mh80v1nzb55r14-python3.6-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/snhqdl4ihcba1cr9ky1rwavlv052mi5q-python3.6-Theano-1.0.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/33fsl0n5nmyiwvkzq4mh80v1nzb55r14-python3.6-libgpuarray-0.7.5.drv', '/nix/store/snhqdl4ihcba1cr9ky1rwavlv052mi5q-python3.6-Theano-1.0.1.drv' failed

GrahamcOfBorg · 2018-04-15T23:27:47Z

Failure on x86_64-darwin (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
r_vree_1.cpp:1732:26: warning: code will never be executed [-Wunreachable-code]
                module = PyImport_ImportModuleLevelObject(
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
building of '/nix/store/x9ff903kpnmmvf39cdw4pbp08rl30mmh-python3.6-Cython-0.28.1.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/33fsl0n5nmyiwvkzq4mh80v1nzb55r14-python3.6-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/snhqdl4ihcba1cr9ky1rwavlv052mi5q-python3.6-Theano-1.0.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/33fsl0n5nmyiwvkzq4mh80v1nzb55r14-python3.6-libgpuarray-0.7.5.drv', '/nix/store/snhqdl4ihcba1cr9ky1rwavlv052mi5q-python3.6-Theano-1.0.1.drv' failed

GrahamcOfBorg · 2018-04-15T23:57:42Z

Failure on x86_64-darwin (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

                module = PyImport_ImportModuleLevelObject(
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
clang-5.0: warning: argument unused during compilation: '-fno-strict-overflow' [-Wunused-command-line-argument]
building of '/nix/store/x9ff903kpnmmvf39cdw4pbp08rl30mmh-python3.6-Cython-0.28.1.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/33fsl0n5nmyiwvkzq4mh80v1nzb55r14-python3.6-libgpuarray-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/snhqdl4ihcba1cr9ky1rwavlv052mi5q-python3.6-Theano-1.0.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/33fsl0n5nmyiwvkzq4mh80v1nzb55r14-python3.6-libgpuarray-0.7.5.drv', '/nix/store/snhqdl4ihcba1cr9ky1rwavlv052mi5q-python3.6-Theano-1.0.1.drv' failed

knedlsepp · 2018-04-16T16:10:29Z

@Mic92: The ofBorg builds timed out, because of an unrelated Cython rebuild. I just rebased locally on upstream/master and could get cython from cache.nixos.org. Would a new "ofBorg build" pick up the now cached cython or do I need to force-push the rebase?

GrahamcOfBorg · 2018-04-17T02:29:44Z

Success on x86_64-linux (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/lib  /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/bin
patching script interpreter paths in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1
/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/lib/python3.6/site-packages/theano/misc/check_blas_many.sh: interpreter directive changed from "/bin/bash" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/bash"
head: error writing 'standard output': Broken pipe
checking for references to /build in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1...
wrapping `/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/bin/theano-nose'...
wrapping `/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/bin/theano-cache'...

GrahamcOfBorg · 2018-04-17T02:31:56Z

Success on x86_64-linux (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

/build/Theano-1.0.1
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/lib  /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/bin
patching script interpreter paths in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1
/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/lib/python3.6/site-packages/theano/misc/check_blas_many.sh: interpreter directive changed from "/bin/bash" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/bash"
checking for references to /build in /nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1...
wrapping `/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/bin/theano-cache'...
wrapping `/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1/bin/theano-nose'...

GrahamcOfBorg · 2018-04-17T02:59:59Z

Success on x86_64-linux (full log)

Attempted: python3Packages.Theano, python2Packages.Theano, python27Packages.libgpuarray, python3Packages.libgpuarray, clblas

Partial log (click to expand)

/nix/store/9af4c4hfrphrqdsl1gad9gss6d8vabfg-python3.6-Theano-1.0.1
/nix/store/jygnxkykpy37dk1vs22wyx0bfk3f2zva-python2.7-Theano-1.0.1
/nix/store/32qadyyrp9r4cm4hwz0zpgdp61ah70ff-python2.7-libgpuarray-0.7.5
/nix/store/l464agyffh5qq2pr1jw92h2fdsx6qc9x-python3.6-libgpuarray-0.7.5
/nix/store/qagkks3v46fha3x0jm198ws359jg6ym5-clblas-2.12

Mic92 · 2018-04-17T12:59:41Z

@LnL7 should be changes like this be backported, if the they fix darwin?

LnL7 · 2018-04-17T18:04:46Z

@Mic92 yes, unsupported packages are usually just never tested so backporting that is pretty safe.

Backported to 18.03 in 3fd2b19.

knedlsepp added 2 commits April 15, 2018 12:12

clblas: Enable darwin build

357d8f6

pythonPackages.libgpuarray: enable darwin build

d96f0c5

knedlsepp requested a review from FRidh as a code owner April 15, 2018 17:35

GrahamcOfBorg added 6.topic: darwin Running or building packages on Darwin 6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Apr 15, 2018

LnL7 approved these changes Apr 15, 2018

View reviewed changes

Mic92 merged commit be5ad65 into NixOS:master Apr 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix clblas and libgpuarray on darwin #38971

Fix clblas and libgpuarray on darwin #38971

knedlsepp commented Apr 15, 2018

Mic92 commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

knedlsepp commented Apr 16, 2018

GrahamcOfBorg commented Apr 17, 2018

GrahamcOfBorg commented Apr 17, 2018

GrahamcOfBorg commented Apr 17, 2018

Mic92 commented Apr 17, 2018

LnL7 commented Apr 17, 2018

Fix clblas and libgpuarray on darwin #38971

Fix clblas and libgpuarray on darwin #38971

Conversation

knedlsepp commented Apr 15, 2018

Motivation for this change

Things done

Mic92 commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

GrahamcOfBorg commented Apr 15, 2018

knedlsepp commented Apr 16, 2018

GrahamcOfBorg commented Apr 17, 2018

GrahamcOfBorg commented Apr 17, 2018

GrahamcOfBorg commented Apr 17, 2018

Mic92 commented Apr 17, 2018

LnL7 commented Apr 17, 2018