Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[18.09] pythonPackages.pyarrow: fix running tests on hydra #49175

Merged
merged 1 commit into from Oct 27, 2018
Merged

[18.09] pythonPackages.pyarrow: fix running tests on hydra #49175

merged 1 commit into from Oct 27, 2018

Conversation

veprbl
Copy link
Member

@veprbl veprbl commented Oct 26, 2018

This replaces standard checkPhase using setup.py that uses pytest-runner
with an explicit call to pytest. One reason to do that is because
setup.py triggers another rebuild when asked to do "test". Another
reason is that there seems to be a conflict between possible imports for
pyarrow: pwd/pyarrow vs $out/pyarrow. By some unknown reason this
triggers an import error on hydra and ofborg, but not on my machine. The
solution here is to remove pwd/pyarrow, keep the tests and use direct
call to pytest (setup.py needs pwd/pyarrow). The added benefit is that
we are now testing what is installed in $out.

(cherry picked from commit 6ae3cb4)

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@veprbl
pythonPackages.pyarrow: fix running tests on hydra
a9ca9bd 
This replaces standard checkPhase using setup.py that uses pytest-runner
with an explicit call to pytest. One reason to do that is because
setup.py triggers another rebuild when asked to do "test". Another
reason is that there seems to be a conflict between possible imports for
pyarrow: `pwd`/pyarrow vs $out/pyarrow. By some unknown reason this
triggers an import error on hydra and ofborg, but not on my machine. The
solution here is to remove `pwd`/pyarrow, keep the tests and use direct
call to pytest (setup.py needs `pwd`/pyarrow). The added benefit is that
we are now testing what is installed in $out.

(cherry picked from commit 6ae3cb4)
@veprbl veprbl requested a review from FRidh as a code owner October 26, 2018 18:14
@veprbl veprbl changed the title pythonPackages.pyarrow: fix running tests on hydra [18.09] pythonPackages.pyarrow: fix running tests on hydra Oct 26, 2018
@veprbl
Copy link
Member Author

veprbl commented Oct 26, 2018

@GrahamcOfBorg build python2Packages.pyarrow python3Packages.pyarrow

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

    sep=r'\s{2,}', index_col=['cut', 'color', 'clarity'], header=0

pyarrow/tests/test_parquet.py::test_backwards_compatible_index_multi_level_some_named
  /private/tmp/nix-build-python3.6-pyarrow-0.9.0.drv-0/apache-arrow-0.9.0/python/pyarrow/tests/test_parquet.py:1633: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
    sep=r'\s{2,}', index_col=['cut', 'color', 'clarity'], header=0

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======== 617 passed, 75 skipped, 3 xfailed, 8 warnings in 16.17 seconds ========
/nix/store/rmdglr02x19bxcq3lxrsl9mgyjj1vk5n-python2.7-pyarrow-0.9.0
/nix/store/fiia4spnbzr6njr902id2m2glpxd33p7-python3.6-pyarrow-0.9.0

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

    sep=r'\s{2,}', index_col=['cut', 'color', 'clarity'], header=0

pyarrow/tests/test_parquet.py::test_backwards_compatible_index_multi_level_some_named
  /build/apache-arrow-0.9.0/python/pyarrow/tests/test_parquet.py:1633: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
    sep=r'\s{2,}', index_col=['cut', 'color', 'clarity'], header=0

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======== 616 passed, 76 skipped, 3 xfailed, 8 warnings in 53.14 seconds ========
/nix/store/k1wmja0l8qbi0d513q0j8di644xgad9k-python2.7-pyarrow-0.9.0
/nix/store/db1nsns6drw3maxlp18j739y5wcj06m2-python3.6-pyarrow-0.9.0

@veprbl veprbl mentioned this pull request Oct 26, 2018
Merged
9 tasks
@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python2Packages.pyarrow, python3Packages.pyarrow

Partial log (click to expand)

  /nix/store/i2a86rn0ashqnsgh0f9gj7kaxb772zw9-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/internals.py:312: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    return self.values[slicer]
  /nix/store/i2a86rn0ashqnsgh0f9gj7kaxb772zw9-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/core/indexes/datetimelike.py:428: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
    result = getitem(key)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
 5 failed, 16897 passed, 1027 skipped, 11896 deselected, 37 xfailed, 2 xpassed, 10 warnings in 644.38 seconds
builder for '/nix/store/4xw8mkbil3jmnjpr51bp78abrr76hk38-python3.6-pandas-0.23.4.drv' failed with exit code 1
cannot build derivation '/nix/store/v7rlc5v95s376aai2j4g5g64535mih0y-python3.6-pyarrow-0.9.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/kw69rfksbcb5q3i8wqv94gndwnvzd1bs-python2.7-pyarrow-0.9.0.drv', '/nix/store/v7rlc5v95s376aai2j4g5g64535mih0y-python3.6-pyarrow-0.9.0.drv' failed

@FRidh FRidh merged commit 6b60c06 into NixOS:release-18.09 Oct 27, 2018
@veprbl veprbl deleted the pr/pyarrow_darwin_hydra_fix_18.09 branch December 1, 2020 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh FRidh is a code owner

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@veprbl @GrahamcOfBorg @FRidh