Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.dask: 2.14.0 -> 2.22.0 #94894

Merged
merged 5 commits into from Aug 9, 2020
Merged

python3Packages.dask: 2.14.0 -> 2.22.0 #94894

merged 5 commits into from Aug 9, 2020

Conversation

jonringer
Copy link
Contributor

@jonringer jonringer commented Aug 7, 2020 •
edited

Motivation for this change

Fix dask for: #94841

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@jonringer jonringer requested a review from FRidh as a code owner August 7, 2020 20:54
@FRidh FRidh self-assigned this Aug 7, 2020
@ofborg ofborg bot requested a review from costrouc August 7, 2020 21:47
@jonringer
Copy link
Contributor Author

with exception to dask, all other packages were broken

@teto
Copy link
Member

teto commented Aug 8, 2020

I started a nixpkgs-review and this breaks datatable. Seems like a daunting task to fix everything though so I see no issue merging this already

builder for '/nix/store/208ks10wakrcw1n7cxi0z6s0bdphy29w-python3.7-datatable-0.10.1.drv' failed with exit code 4; last 10 log lines:
  ImportError while loading conftest '/build/datatable-0.10.1/tests/conftest.py'.
  tests/__init__.py:13: in <module>
      from datatable.internal import frame_columns_virtual, frame_integrity_check
  /nix/store/q83m72x8qs0wgc21cj05hv481qxr1mbv-python3.7-datatable-0.10.1/lib/python3.7/site-packages/datatable/__init__.py:24: in <module>
      from .frame import Frame
  /nix/store/q83m72x8qs0wgc21cj05hv481qxr1mbv-python3.7-datatable-0.10.1/lib/python3.7/site-packages/datatable/frame.py:23: in <module>
      from datatable.lib._datatable import Frame
  /nix/store/q83m72x8qs0wgc21cj05hv481qxr1mbv-python3.7-datatable-0.10.1/lib/python3.7/site-packages/datatable/lib/__init__.py:31: in <module>
      from . import _datatable as core
  E   ImportError: cannot import name '_datatable' from 'datatable.lib' (/nix/store/q83m72x8qs0wgc21cj05hv481qxr1mbv-python3.7-datatable-0.10.1/lib/python3.7/site-packages/datatable/lib/__init__.py)
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/9z7bc6bqlwmqzlrhxydzc972lvqz65yl-python3.7-xgboost-0.90.drv' failed to build
note: keeping build directory '/tmp/nix-build-python3.8-datatable-0.10.1.drv-0'
error: --- Error --- nix-daemon
builder for '/nix/store/96k410ck2likby2mcpmp5sh6df6q28lk-python3.8-datatable-0.10.1.drv' failed with exit code 4; last 10 log lines:
  ImportError while loading conftest '/build/datatable-0.10.1/tests/conftest.py'.
  tests/__init__.py:13: in <module>
      from datatable.internal import frame_columns_virtual, frame_integrity_check
  /nix/store/j98mxf1r2hakxvvy94sf0hh4kjq78kdv-python3.8-datatable-0.10.1/lib/python3.8/site-packages/datatable/__init__.py:24: in <module>
      from .frame import Frame
  /nix/store/j98mxf1r2hakxvvy94sf0hh4kjq78kdv-python3.8-datatable-0.10.1/lib/python3.8/site-packages/datatable/frame.py:23: in <module>
      from datatable.lib._datatable import Frame
  /nix/store/j98mxf1r2hakxvvy94sf0hh4kjq78kdv-python3.8-datatable-0.10.1/lib/python3.8/site-packages/datatable/lib/__init__.py:31: in <module>
      from . import _datatable as core
  E   ImportError: cannot import name '_datatable' from partially initialized module 'datatable.lib' (most likely due to a circular import) (/nix/store/j98mxf1r2hakxvvy94sf0hh4kjq78kdv-python3.8-datatable-0.10.1/lib/python3.8/site-packages/datatable/lib/__init__.py)

@jonringer
Copy link
Contributor Author

I started a nixpkgs-review and this breaks datatable. Seems like a daunting task to fix everything though so I see no issue merging this already

it's already broken on master, and has been broken for the past 2 months https://hydra.nixos.org/build/124757737

@jonringer
Copy link
Contributor Author

@costrouc , as most of these are packages that you maintain, do you mind reviewing the changes.

@costrouc
Copy link
Member

costrouc commented Aug 8, 2020

@jonringer I would love to help with this review and also help with in general the scientific/data python stack maintinance. For me the hardest part is knowing how I should and can help with keeping these packages up to date and cleaning them up. Should I just follow e.g. https://hydra.nixos.org/eval/1605243?filter=python38Packages&compare=1605209&full=#tabs-still-fail? What is the way you would recommend way of maintaining? At this point my github notifications are useless since I have way too many notifications (or I just need to be using a better tool for notifications). Also I realize that I contribute a lot of python packages to nixpkgs which does cause a maintenance burden and I would like to help with that.

@costrouc
Copy link
Member

costrouc commented Aug 8, 2020

Result of nixpkgs-review pr 94894 1

3 packages failed to build:
- python37Packages.dask-xgboost
- python37Packages.mask-rcnn
- python38Packages.dask-xgboost
53 packages built:
- caffe
- python37Packages.aplpy
- python37Packages.batchgenerators
- python37Packages.caffe
- python37Packages.dask
- python37Packages.dask-glm
- python37Packages.dask-image
- python37Packages.dask-jobqueue
- python37Packages.dask-ml
- python37Packages.dask-mpi
- python37Packages.datashader
- python37Packages.distributed
- python37Packages.glymur
- python37Packages.image-match
- python37Packages.imagecorruptions
- python37Packages.imgaug
- python37Packages.intake
- python37Packages.pims
- python37Packages.pyfftw
- python37Packages.scikitimage
- python37Packages.sparse
- python37Packages.spectral-cube
- python37Packages.streamz
- python37Packages.stumpy
- python37Packages.stytra
- python37Packages.sunpy
- python37Packages.tensorly
- python38Packages.aplpy
- python38Packages.batchgenerators
- python38Packages.caffe
- python38Packages.dask
- python38Packages.dask-glm
- python38Packages.dask-image
- python38Packages.dask-jobqueue
- python38Packages.dask-ml
- python38Packages.dask-mpi
- python38Packages.datashader
- python38Packages.distributed
- python38Packages.glymur
- python38Packages.image-match
- python38Packages.imagecorruptions
- python38Packages.imgaug
- python38Packages.intake
- python38Packages.pims
- python38Packages.pyfftw
- python38Packages.scikitimage
- python38Packages.sparse
- python38Packages.spectral-cube
- python38Packages.streamz
- python38Packages.stumpy
- python38Packages.stytra
- python38Packages.sunpy
- python38Packages.tensorly

@jonringer
Copy link
Contributor Author

@costrouc we don't have a good mechanism to notify build failures yet

Temporary solution, create a .nix file with the packages you care about, then attempt to build them periodically:

$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz nix build -f packages.nix
[18.8 MiB DL]
[15:25:30] jon@jon-desktop /home/jon/projects/nixpkgs (bzip2-1.0.8-staging)
$ cat packages.nix
with import <nixpkgs> { };

[
  python3Packages.dask

]

@jonringer
Copy link
Contributor Author

this is a significant improvement, going to merge

@jonringer jonringer merged commit bba261c into NixOS:master Aug 9, 2020
@jonringer jonringer deleted the bump-dask branch August 9, 2020 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

@costrouc costrouc Awaiting requested review from costrouc

Assignees

@FRidh FRidh

Labels
6.topic: python 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jonringer @teto @costrouc @FRidh