pythonPackages.pyarrow: fix running tests on hydra #49130

veprbl · 2018-10-26T02:54:51Z

This replaces standard checkPhase using setup.py that uses pytest-runner
with an explicit call to pytest. One reason to do that is because
setup.py triggers another rebuild when asked to do "test". Another
reason is that there seems to be a conflict between possible imports for
pyarrow: `pwd`/pyarrow vs $out/pyarrow. By some unknown reason this
triggers an import error on hydra and ofborg, but not on my machine. The
solution here is to remove `pwd`/pyarrow, keep the tests and use direct
call to pytest (setup.py needs `pwd`/pyarrow). The added benefit is that
we are now testing what is installed in $out.

Motivation for this change

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

veprbl · 2018-10-26T02:55:26Z

@GrahamcOfBorg build python2Packages.pyarrow python3Packages.pyarrow

GrahamcOfBorg · 2018-10-26T02:57:38Z

Failure on x86_64-darwin (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

    if left_value != right_value:

tests/test_parquet.py::test_coerce_timestamps
  /nix/store/db90qrmip6s01fp8sd6chv1lmg68a4iw-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/dtypes/missing.py:431: FutureWarning: In the future, NAT != NAT will be True rather than False.
    if left_value != right_value:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
== 1031 passed, 101 skipped, 6 xfailed, 3 warnings, 72 error in 34.50 seconds ==
builder for '/nix/store/rn19axxdk88ls33sw7wimrbgmq4f5fpl-python3.6-pyarrow-0.11.0.drv' failed with exit code 1
error: build of '/nix/store/9lxgvwz4ndni1nqwq7mf5wbvjr808g1f-python2.7-pyarrow-0.11.0.drv', '/nix/store/rn19axxdk88ls33sw7wimrbgmq4f5fpl-python3.6-pyarrow-0.11.0.drv' failed

GrahamcOfBorg · 2018-10-26T03:01:29Z

Failure on x86_64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

    if left_value != right_value:

tests/test_parquet.py::test_coerce_timestamps
  /nix/store/2x3yfw2fyyp8gq52v0wa13p75lbwxqxd-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/dtypes/missing.py:431: FutureWarning: In the future, NAT != NAT will be True rather than False.
    if left_value != right_value:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
== 1033 passed, 99 skipped, 6 xfailed, 3 warnings, 72 error in 74.25 seconds ===
builder for '/nix/store/pm4rpd017hm0hh7j2fm5k981301qdl19-python3.6-pyarrow-0.11.0.drv' failed with exit code 1
error: build of '/nix/store/fwyc9dr9g62pwgazgh4vdyrblsib0hvq-python2.7-pyarrow-0.11.0.drv', '/nix/store/pm4rpd017hm0hh7j2fm5k981301qdl19-python3.6-pyarrow-0.11.0.drv' failed

veprbl · 2018-10-26T03:01:44Z

Right, forgot that I was testing against release-18.09

@GrahamcOfBorg build python2Packages.pyarrow python3Packages.pyarrow

GrahamcOfBorg · 2018-10-26T03:03:08Z

Failure on x86_64-darwin (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

pyarrow/tests/test_ipc.py::test_pandas_serialize_round_trip_multi_index PASSED [ 61%]
pyarrow/tests/test_ipc.py::test_serialize_pandas_empty_dataframe PASSED  [ 61%]
pyarrow/tests/test_ipc.py::test_pandas_serialize_round_trip_not_string_columns PASSED [ 61%]
pyarrow/tests/test_ipc.py::test_serialize_pandas_no_preserve_index PASSED [ 61%]
pyarrow/tests/test_ipc.py::test_serialize_with_pandas_objects PASSED     [ 61%]
pyarrow/tests/test_ipc.py::test_schema_batch_serialize_methods PASSED    [ 61%]
pyarrow/tests/test_ipc.py::test_schema_serialization_with_metadata PASSED [ 62%]
pyarrow/tests/test_jvm.py::test_jvm_buffer
builder for '/nix/store/v0qh415rbcmjpqkgkz4appk0z18jg861-python3.6-pyarrow-0.11.0.drv' failed with exit code 97
error: build of '/nix/store/dfhgzrk4iksj9w9574f3b1icw8dmz8dz-python2.7-pyarrow-0.11.0.drv', '/nix/store/v0qh415rbcmjpqkgkz4appk0z18jg861-python3.6-pyarrow-0.11.0.drv' failed

GrahamcOfBorg · 2018-10-26T03:05:57Z

Failure on x86_64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

/nix/store/0m03fcr9sxgpm0njigjb6yicnd66y9ls-python2.7-JPype1-0.6.3/lib/python2.7/site-packages/jpype/_jvmfinder.py:97: OSError
=============================== warnings summary ===============================
pyarrow/tests/test_convert_pandas.py::TestListTypes::()::test_column_of_arrays
  /nix/store/xmj7jj18l6lvwpn5knl0vbxa2j3r0fj9-python2.7-pandas-0.23.4/lib/python2.7/site-packages/pandas/core/dtypes/missing.py:431: FutureWarning: In the future, NAT != NAT will be True rather than False.
    if left_value != right_value:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
== 1030 passed, 102 skipped, 6 xfailed, 1 warnings, 72 error in 22.83 seconds ==
builder for '/nix/store/75ynn816d20s4yps442ad0ix7ak7qw6p-python2.7-pyarrow-0.11.0.drv' failed with exit code 1
error: build of '/nix/store/75ynn816d20s4yps442ad0ix7ak7qw6p-python2.7-pyarrow-0.11.0.drv', '/nix/store/9704w5q94686xz8jq957myz6fkzfyfaj-python3.6-pyarrow-0.11.0.drv' failed

This replaces standard checkPhase using setup.py that uses pytest-runner with an explicit call to pytest. One reason to do that is because setup.py triggers another rebuild when asked to do "test". Another reason is that there seems to be a conflict between possible imports for pyarrow: `pwd`/pyarrow vs $out/pyarrow. By some unknown reason this triggers an import error on hydra and ofborg, but not on my machine. The solution here is to remove `pwd`/pyarrow, keep the tests and use direct call to pytest (setup.py needs `pwd`/pyarrow). The added benefit is that we are now testing what is installed in $out.

veprbl · 2018-10-26T03:15:11Z

@GrahamcOfBorg build python2Packages.pyarrow python3Packages.pyarrow

GrahamcOfBorg · 2018-10-26T03:16:40Z

Success on x86_64-darwin (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)


pyarrow/tests/test_parquet.py::test_coerce_timestamps
  /nix/store/db90qrmip6s01fp8sd6chv1lmg68a4iw-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/dtypes/missing.py:431: FutureWarning: In the future, NAT != NAT will be True rather than False.
    if left_value != right_value:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======= 1031 passed, 102 skipped, 5 xfailed, 3 warnings in 20.41 seconds =======
pytestcachePhase
/nix/store/axa0klbnbib98s9f2ddsg4g8vzqckzsr-python2.7-pyarrow-0.11.0
/nix/store/hvws4j5crs2jq4pm6cgfvs0p6j2ivy5x-python3.6-pyarrow-0.11.0

GrahamcOfBorg · 2018-10-26T03:19:37Z

Success on x86_64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)


pyarrow/tests/test_parquet.py::test_coerce_timestamps
  /nix/store/2x3yfw2fyyp8gq52v0wa13p75lbwxqxd-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/dtypes/missing.py:431: FutureWarning: In the future, NAT != NAT will be True rather than False.
    if left_value != right_value:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======= 1033 passed, 100 skipped, 5 xfailed, 3 warnings in 64.29 seconds =======
pytestcachePhase
/nix/store/xjglr04p853jbm1nhljhwv06qacrfzxh-python2.7-pyarrow-0.11.0
/nix/store/wiayndfdg722fvvmj27pi4kryixlb13j-python3.6-pyarrow-0.11.0

GrahamcOfBorg · 2018-10-26T03:20:40Z

Failure on aarch64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

  /nix/store/qfdbglqq4rmjv33y3nhi5qlifgxyg34m-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/indexes/datetimelike.py:428: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    result = getitem(key)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
 5 failed, 16897 passed, 1027 skipped, 11896 deselected, 37 xfailed, 2 xpassed, 10 warnings in 597.66 seconds
builder for '/nix/store/sfgg7h8zc274x7n3c4v97bjgsv4djrbz-python2.7-pandas-0.23.4.drv' failed with exit code 1
cannot build derivation '/nix/store/k4i86p6gfimmzm1akwsyqv15kj69g5h2-python2.7-pyarrow-0.11.0.drv': 1 dependencies couldn't be built
builder for '/nix/store/k37vq3id2j6pmwr9dc64gghdk5yidw7x-python3.6-pandas-0.23.4.drv' failed with exit code 1
cannot build derivation '/nix/store/6js714ldas0wjz9c744slibdgnjgsya2-python3.6-pyarrow-0.11.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/6js714ldas0wjz9c744slibdgnjgsya2-python3.6-pyarrow-0.11.0.drv', '/nix/store/k4i86p6gfimmzm1akwsyqv15kj69g5h2-python2.7-pyarrow-0.11.0.drv' failed

veprbl · 2018-10-26T03:21:23Z

Success!

GrahamcOfBorg · 2018-10-26T03:45:25Z

Failure on aarch64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

  /nix/store/qfdbglqq4rmjv33y3nhi5qlifgxyg34m-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/internals.py:312: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    return self.values[slicer]
  /nix/store/qfdbglqq4rmjv33y3nhi5qlifgxyg34m-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/indexes/datetimelike.py:428: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    result = getitem(key)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
 5 failed, 16897 passed, 1027 skipped, 11896 deselected, 37 xfailed, 2 xpassed, 10 warnings in 597.40 seconds
builder for '/nix/store/k37vq3id2j6pmwr9dc64gghdk5yidw7x-python3.6-pandas-0.23.4.drv' failed with exit code 1
cannot build derivation '/nix/store/p50lmab3y5py1gipsnsi5c7pp2maljzq-python3.6-pyarrow-0.11.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/lbz8mrscn3p56cw2vz5fdlf46z2i185c-python2.7-pyarrow-0.11.0.drv', '/nix/store/p50lmab3y5py1gipsnsi5c7pp2maljzq-python3.6-pyarrow-0.11.0.drv' failed

GrahamcOfBorg · 2018-10-26T04:10:14Z

Failure on aarch64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

  /nix/store/qfdbglqq4rmjv33y3nhi5qlifgxyg34m-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/internals.py:312: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    return self.values[slicer]
  /nix/store/qfdbglqq4rmjv33y3nhi5qlifgxyg34m-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/indexes/datetimelike.py:428: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    result = getitem(key)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
 5 failed, 16897 passed, 1027 skipped, 11896 deselected, 37 xfailed, 2 xpassed, 10 warnings in 593.53 seconds
builder for '/nix/store/k37vq3id2j6pmwr9dc64gghdk5yidw7x-python3.6-pandas-0.23.4.drv' failed with exit code 1
cannot build derivation '/nix/store/k1iwyjhi2xlr7l3sb81m4xy94b7rpagr-python3.6-pyarrow-0.11.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/k1iwyjhi2xlr7l3sb81m4xy94b7rpagr-python3.6-pyarrow-0.11.0.drv', '/nix/store/s6z68lma527d6hd5pwq33z68s8001fr2-python2.7-pyarrow-0.11.0.drv' failed

Mic92 · 2018-10-26T08:18:51Z

Does this affect 18.09?

veprbl · 2018-10-26T18:21:14Z

@Mic92 Yes, I opened #49175

veprbl requested a review from FRidh as a code owner October 26, 2018 02:54

GrahamcOfBorg added the 6.topic: python label Oct 26, 2018

veprbl force-pushed the pr/pyarrow_darwin_hydra_fix branch from 0b652c4 to b4306e0 Compare October 26, 2018 03:01

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Oct 26, 2018

veprbl added 2 commits October 25, 2018 23:12

pyarrow: disable jvm test

e76082a

veprbl force-pushed the pr/pyarrow_darwin_hydra_fix branch from b4306e0 to e76082a Compare October 26, 2018 03:14

Mic92 merged commit e438125 into NixOS:master Oct 26, 2018

veprbl deleted the pr/pyarrow_darwin_hydra_fix branch December 1, 2020 16:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.pyarrow: fix running tests on hydra #49130

pythonPackages.pyarrow: fix running tests on hydra #49130

veprbl commented Oct 26, 2018 •

edited

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

Mic92 commented Oct 26, 2018

veprbl commented Oct 26, 2018

pythonPackages.pyarrow: fix running tests on hydra #49130

pythonPackages.pyarrow: fix running tests on hydra #49130

Conversation

veprbl commented Oct 26, 2018 • edited

Motivation for this change

Things done

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

veprbl commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

GrahamcOfBorg commented Oct 26, 2018

Mic92 commented Oct 26, 2018

veprbl commented Oct 26, 2018

veprbl commented Oct 26, 2018 •

edited