python36Packages.cython: Disable tests on aarch64 #41453

etu · 2018-06-04T15:21:41Z

Motivation for this change

The tests doesn't run on aarch64 and it's reported to upstream here:
cython/cython#2308

This would also resolve #41347 for now.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

cc @FRidh @dezgeg

GrahamcOfBorg · 2018-06-04T15:35:02Z

Success on x86_64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

these paths will be fetched (2.74 MiB download, 12.31 MiB unpacked):
  /nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2
copying path '/nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2' from 'https://cache.nixos.org'...
/nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2

FRidh

Disable specific tests instead of the whole suite.

GrahamcOfBorg · 2018-06-04T15:47:42Z

Success on aarch64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

shrinking /nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/lib/python3.6/site-packages/Cython/Tempita/_tempita.cpython-36m-aarch64-linux-gnu.so
shrinking /nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/lib/python3.6/site-packages/Cython/StringIOTree.cpython-36m-aarch64-linux-gnu.so
strip is /nix/store/qg2agrqkf240s656d207zqhipl0bc2id-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/lib  /nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/bin
patching script interpreter paths in /nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2
checking for references to /build in /nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2...
wrapping `/nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/bin/cythonize'...
wrapping `/nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/bin/cython'...
wrapping `/nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2/bin/cygdb'...
/nix/store/8p0bgxylavg1l4r3wd0wwmrhkmp2n4dq-python3.6-Cython-0.28.2

etu · 2018-06-04T17:08:24Z

@FRidh I'm not sure how python behaves if getting two --exclude flags though.

GrahamcOfBorg · 2018-06-04T18:07:18Z

Success on x86_64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)


OK
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
ALL DONE
/nix/store/69lq0i8zjm0fviwp02lxc8grpsr2q740-python3.6-Cython-0.28.2

GrahamcOfBorg · 2018-06-04T18:21:58Z

Failure on aarch64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

                 from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from memoryviewattrs.cpp:580:
/nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
acquired default
acquired Global_A
building of '/nix/store/7g7hv0caz1wr66ca0r4yj2v867mnghd6-python3.6-Cython-0.28.2.drv' timed out after 3600 seconds
error: build of '/nix/store/7g7hv0caz1wr66ca0r4yj2v867mnghd6-python3.6-Cython-0.28.2.drv' failed

dotlambda · 2018-06-05T07:50:21Z

pkgs/development/python-modules/Cython/default.nix

  checkPhase = ''
    export HOME="$NIX_BUILD_TOP"
    ${python.interpreter} runtests.py \
-      ${if stdenv.cc.isClang or false then ''--exclude="(cpdef_extern_func|libcpp_algo)"'' else ""}
+      ${if stdenv.cc.isClang or false then ''--exclude="(cpdef_extern_func|libcpp_algo)"'' else ""} \


lib.optionalString

veprbl · 2018-06-05T18:52:55Z

pkgs/development/python-modules/Cython/default.nix

  checkPhase = ''
    export HOME="$NIX_BUILD_TOP"
    ${python.interpreter} runtests.py \
-      ${if stdenv.cc.isClang or false then ''--exclude="(cpdef_extern_func|libcpp_algo)"'' else ""}
+      ${if stdenv.cc.isClang or false then ''--exclude="(cpdef_extern_func|libcpp_algo)"'' else ""} \
+      ${if stdenv.isArch64 or false then ''--exclude="test_coerce_to_numpy"''  else ""}


No or false here - stdenv.isArch64 is always defined.

also s/isArch64/isAarch64/

To not break compilation with clang on aarch64 we could refactor to something like

let excluded_tests = [] ++ stdenv.lib.optionals (stdenv.cc.isClang or false) [ "cpdef_extern_func" "libcpp_algo" ] ++ stdenv.lib.optionals stdenv.isAarch64 [ "test_coerce_to_numpy" ] ; in stdenv.lib.optionalString (builtins.length excluded_tests != 0) ''--exclude="(${builtins.concatStringsSep "|" excluded_tests})"''

(IIRC two exclude flags won't work here)

stdenv.cc.isClang isn't always defined?

@dotlambda When I was working with it, it was only defined for clang.cc. I don't think anything changed since then.

Actually, it appears, multiple exclude options are allowed: cython/cython@7631e5d

GrahamcOfBorg · 2018-06-05T20:06:34Z

Success on x86_64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

these paths will be fetched (13.22 MiB download, 69.07 MiB unpacked):
  /nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2
  /nix/store/acxw8n8pijhy9zl2ghhr0ihwk31ry6wq-python3.6-setuptools-39.0.1
  /nix/store/d0az5vfrk5cm65wcvxmfj2hjk613k9ni-xz-5.2.4
  /nix/store/rsa1rsd6c6szy0dmi61599b33aqiy269-python3-3.6.5
copying path '/nix/store/d0az5vfrk5cm65wcvxmfj2hjk613k9ni-xz-5.2.4' from 'https://cache.nixos.org'...
copying path '/nix/store/rsa1rsd6c6szy0dmi61599b33aqiy269-python3-3.6.5' from 'https://cache.nixos.org'...
copying path '/nix/store/acxw8n8pijhy9zl2ghhr0ihwk31ry6wq-python3.6-setuptools-39.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2' from 'https://cache.nixos.org'...
/nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2

dotlambda · 2018-06-05T20:08:29Z

pkgs/development/python-modules/Cython/default.nix

+    ++ stdenv.lib.optionals (stdenv.cc.isClang or false) [ "cpdef_extern_func" "libcpp_algo" ]
+    # Some tests in the test suite isn't working on aarch64. Disable them for
+    # now until upstream finds a workaround.
+    ++ stdenv.lib.optionals stdenv.isAarch64 [ "test_coerce_to_numpy" ]


Please add a link to the upstream issue.

GrahamcOfBorg · 2018-06-05T20:33:33Z

Success on x86_64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

/nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2

GrahamcOfBorg · 2018-06-05T21:06:02Z

Failure on aarch64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

                 from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from memoryviewattrs.cpp:580:
/nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
acquired default
acquired Global_A
building of '/nix/store/qz8a66gp92hn3v3aajplyx4r91m07kj5-python3.6-Cython-0.28.2.drv' timed out after 3600 seconds
error: build of '/nix/store/qz8a66gp92hn3v3aajplyx4r91m07kj5-python3.6-Cython-0.28.2.drv' failed

GrahamcOfBorg · 2018-06-05T22:06:07Z

Failure on aarch64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

   int __pyx_t_2[1];
       ^~~~~~~~~
==============================

=== C/C++ compiler error output: ===
khavkine1.cpp: In function ‘PyObject* __pyx_f_9khavkine1___pyx_unpickle_T__set_state(__pyx_obj_9khavkine1_T*, PyObject*)’:
khavkine1.cpp:1914:7: warning: ‘__pyx_t_2’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   int __pyx_t_2[1];
building of '/nix/store/qz8a66gp92hn3v3aajplyx4r91m07kj5-python3.6-Cython-0.28.2.drv' timed out after 3600 seconds
error: build of '/nix/store/qz8a66gp92hn3v3aajplyx4r91m07kj5-python3.6-Cython-0.28.2.drv' failed

etu · 2018-06-06T06:09:40Z

I just want to wait for my raspberry to finish building this just to be sure... :)

…arch64

GrahamcOfBorg · 2018-06-06T12:06:47Z

Success on x86_64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

/nix/store/1lv0f3c2dqzrxsgvyh3hzggk0kd1pf0l-python3.6-Cython-0.28.2

GrahamcOfBorg · 2018-06-06T13:06:59Z

Failure on aarch64-linux (full log)

Attempted: python36Packages.cython

Partial log (click to expand)

  ^~~~~~~
In file included from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1816:0,
                 from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from memoryviewattrs.cpp:580:
/nix/store/sh7z6vib9kxhsj5rjydcd0hqs9acs5y0-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
building of '/nix/store/zh4vgh550i8iypbhi305wavx1xdkcrv2-python3.6-Cython-0.28.2.drv' timed out after 3600 seconds
error: build of '/nix/store/zh4vgh550i8iypbhi305wavx1xdkcrv2-python3.6-Cython-0.28.2.drv' failed

etu · 2018-06-07T06:14:56Z

I have just confirmed that the current commit does build on my raspberry pi 3.

But this bit is a bit worrying to me, I will ask about it upstream as well.

----------------------------------------------------------------------
Ran 11199 tests in 52068.367s

OK

That it takes 14.5 hours to run the tests is a bit much maybe? Building the program takes about an hour.

dotlambda · 2018-06-07T08:49:04Z

Let's merge this anyway so we can find out how long it takes Hydra to build. Should be a lot quicker than on a Raspberry.

dotlambda · 2018-06-07T09:21:10Z

also few tests failed on i686-linux

If you know which ones fail, please open a PR disabling them on i686.

etu · 2018-06-07T09:43:18Z

@volth When you know this, you should report it upstream as well :)

etu · 2018-06-08T05:45:24Z

@dotlambda Seems to timeout after 10 hours: https://hydra.nixos.org/build/75636013 😞

etu requested a review from FRidh as a code owner June 4, 2018 15:21

GrahamcOfBorg added 6.topic: python 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jun 4, 2018

FRidh requested changes Jun 4, 2018

View reviewed changes

etu force-pushed the disable-cython-tests-on-aarch64 branch from 08af4d4 to aa92ba9 Compare June 4, 2018 17:07

etu force-pushed the disable-cython-tests-on-aarch64 branch from aa92ba9 to bf9a70d Compare June 4, 2018 17:09

GrahamcOfBorg added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jun 4, 2018

dotlambda reviewed Jun 5, 2018

View reviewed changes

veprbl reviewed Jun 5, 2018

View reviewed changes

etu force-pushed the disable-cython-tests-on-aarch64 branch from bf9a70d to acb0de9 Compare June 5, 2018 19:51

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 and removed 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Jun 5, 2018

dotlambda reviewed Jun 5, 2018

View reviewed changes

etu force-pushed the disable-cython-tests-on-aarch64 branch from acb0de9 to 59ebe3b Compare June 5, 2018 20:09

python36Packages.cython: Disable the "test_coerce_to_numpy" test on a…

370f41b

…arch64

etu force-pushed the disable-cython-tests-on-aarch64 branch from 59ebe3b to 370f41b Compare June 6, 2018 11:41

dotlambda merged commit df92030 into NixOS:master Jun 7, 2018

etu deleted the disable-cython-tests-on-aarch64 branch June 7, 2018 08:53

etu mentioned this pull request Jun 8, 2018

Issues building python36Packages.cython on hydra on aarch64 #41673

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python36Packages.cython: Disable tests on aarch64 #41453

python36Packages.cython: Disable tests on aarch64 #41453

etu commented Jun 4, 2018

GrahamcOfBorg commented Jun 4, 2018

FRidh left a comment

GrahamcOfBorg commented Jun 4, 2018

etu commented Jun 4, 2018

GrahamcOfBorg commented Jun 4, 2018

GrahamcOfBorg commented Jun 4, 2018

dotlambda Jun 5, 2018

veprbl Jun 5, 2018

veprbl Jun 5, 2018

veprbl Jun 5, 2018 •

edited

dotlambda Jun 5, 2018

veprbl Jun 5, 2018

veprbl Jun 7, 2018

GrahamcOfBorg commented Jun 5, 2018

dotlambda Jun 5, 2018

etu Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

etu commented Jun 6, 2018

GrahamcOfBorg commented Jun 6, 2018

GrahamcOfBorg commented Jun 6, 2018

etu commented Jun 7, 2018

dotlambda commented Jun 7, 2018

dotlambda commented Jun 7, 2018

etu commented Jun 7, 2018

etu commented Jun 8, 2018

python36Packages.cython: Disable tests on aarch64 #41453

python36Packages.cython: Disable tests on aarch64 #41453

Conversation

etu commented Jun 4, 2018

Motivation for this change

Things done

GrahamcOfBorg commented Jun 4, 2018

FRidh left a comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 4, 2018

etu commented Jun 4, 2018

GrahamcOfBorg commented Jun 4, 2018

GrahamcOfBorg commented Jun 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

veprbl Jun 5, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

etu commented Jun 6, 2018

GrahamcOfBorg commented Jun 6, 2018

GrahamcOfBorg commented Jun 6, 2018

etu commented Jun 7, 2018

dotlambda commented Jun 7, 2018

dotlambda commented Jun 7, 2018

etu commented Jun 7, 2018

etu commented Jun 8, 2018

veprbl Jun 5, 2018 •

edited