pythonPackages.distributed: 1.15.1 -> 1.22.1 #44657

costrouc · 2018-08-08T02:27:07Z

Motivation for this change

Wanted to run dask distributed on python 2.7 and 3+. Additionally wanted all tests to pass. A lot of work had to be put into adding testing frameworks and ensuring that all the underlying libraries were also compatible.

Things done

For all the following all tests pass and run on python 2.7, 3+ now.

pythonPackages.distributed: 1.15.1 -> 1.22.1 and refactor
pythonPackages.pytest-repeat: init at 0.6.0
pythonPackages.joblib: 0.21.1 -> 0.21.2 and refactor
pythonPackages.python-lz4: init at 2.1.0
pythonPackages.pytest-faulthandler: init at 1.5.0
pythonPackages.pytest-timeout: refactor

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
[] Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

costrouc · 2018-08-08T02:28:03Z

Requires myself to be added to maintainers (pull request #44655) list before this pull request

xeji · 2018-08-08T06:08:39Z

@GrahamcOfBorg eval

dotlambda · 2018-08-08T08:39:00Z

pkgs/development/python-modules/distributed/default.nix

+
+  # py.test not picking up local config file, even when running
+  # manually: E ValueError: no option named '--runslow'
+  doCheck = false;


Maybe you need to add pytest-repeat or pytest-faulthandler to checkInputs.

See https://github.com/dask/distributed/blob/master/continuous_integration/travis/install.sh#L57

costrouc · 2018-08-08T19:52:53Z

@dotlambda I would really like to thank you for how detailed you have been in your code reviews. I have learned a lot on how to properly package with nix and several git techniques I didn't know how to do (squash, reset --soft, etc.) really helped me with learning how to create good looking commits.

There are a lot of packages updated in the following commit. It was necessary in order to make all the tests run and ensure that it also worked on python 2.7. Looks like 2.7 was neglected in some of the packages that I have refactored along with tests.

Some questions I have about style.

is the goal to eventually move all packages out of pkgs/top-level/python-packages.nix -> pkgs/development/python-modules/...?
pypi archives most of the time do not have all the tests and setup. Is it good form to use the github repository instead? At least here we can run all the tests

dotlambda · 2018-08-08T20:16:38Z

is the goal to eventually move all packages out of pkgs/top-level/python-packages.nix -> pkgs/development/python-modules/...?

Yes, and I think more than 2/3 have already been moved.

pypi archives most of the time do not have all the tests and setup. Is it good form to use the github repository instead? At least here we can run all the tests

Fetching from GitHub has the disadvantage that our batch upgrade script won't work until #41149 is done. I sometimes open upstream PRs to add tests to the tarball, e.g. ethereum/eth-hash#20.
EDIT: However, if you prefer to do so and intend to maintain the packages, there's no problem with using fetchFromGitHub. For the packages that I'm maintaining, I usually add GitHub's releases.atom to my RSS reader.

dotlambda · 2018-08-08T20:20:20Z

pkgs/development/python-modules/pytest-timeout/default.nix

+  # remove patch after version 1.3.1
+  # assuming they fix installation on python 2.7 and 3.6
+  # they have encoding issue
+  patches = [ ./0001-open-encoding.patch ];


I think it's best to use fetchpatch with https://bitbucket.org/pytest-dev/pytest-timeout/commits/9de81d3fc57a71a36d418d4aa181c241a7c5350f/raw.

So, import io and io.open() don't work?

Turns out that I was not including io in open yes that patch works I will add this. Very cool to see that you can do patches with urls. Thanks!

A big advantage of using fetchpatch is that it saves disk space on every checkout/download of Nixpkgs, i.e. on every computer of a Nix user.

dotlambda · 2018-08-08T20:22:17Z

@GrahamcOfBorg build python2.pkgs.pytest-timeout python3.pkgs.pytest-timeout python2.pkgs.pytest-faulthandler python3.pkgs.pytest-faulthandler

dotlambda · 2018-08-08T20:22:41Z

pkgs/development/python-modules/python-lz4/default.nix

+  # get full repository inorder to run tests
+  src = fetchFromGitHub {
+    owner = "python-lz4";
+    repo = "${pname}";


repo = pname

dotlambda · 2018-08-08T20:23:09Z

pkgs/development/python-modules/python-lz4/default.nix

+
+  buildInputs = [ setuptools_scm pkgconfig ];
+  checkInputs = [ pytest psutil ];
+  propagatedBuildInputs = [] ++ lib.optional (pythonOlder "3.0") future;


propagatedBuildInputs = lib.optionals (!isPy3k) future;

GrahamcOfBorg · 2018-08-08T20:23:23Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: python2.pkgs.pytest-timeout, python3.pkgs.pytest-timeout, python2.pkgs.pytest-faulthandler, python3.pkgs.pytest-faulthandler

Partial log (click to expand)

Can't start up: not enough memory

Cannot nix-instantiate `python2.pkgs.pytest-faulthandler' because:
GC Warning: Failed to expand heap by 10737418240 bytes
Can't start up: not enough memory

Cannot nix-instantiate `python3.pkgs.pytest-faulthandler' because:
GC Warning: Failed to expand heap by 10737418240 bytes
Can't start up: not enough memory

GrahamcOfBorg · 2018-08-08T20:24:06Z

Success on x86_64-darwin (full log)

Attempted: python2.pkgs.pytest-timeout, python3.pkgs.pytest-timeout, python2.pkgs.pytest-faulthandler, python3.pkgs.pytest-faulthandler

Partial log (click to expand)

test_pytest_timeout.py ...................................               [100%]

========================== 35 passed in 52.28 seconds ==========================
test_pytest_timeout.py ...................................               [100%]

========================== 35 passed in 54.43 seconds ==========================
/nix/store/rplfrxvynrzr52fvlh84n2alflm0n9i8-python2.7-pytest-timeout-1.3.1
/nix/store/10fm64nzifdpbx05ch13h4wl3s24wjg4-python3.6-pytest-timeout-1.3.1
/nix/store/rymlxx9b5d3lsdvqb6lkdn8pdzaiv6kp-python2.7-pytest-faulthandler-1.5.0
/nix/store/cy1a9fzh5nvlf8bmsy0fbkyg1szl67zd-python3.6-pytest-faulthandler-1.5.0

dotlambda · 2018-08-08T20:26:38Z

pkgs/development/python-modules/pytest-repeat/default.nix

+  '';
+
+  meta = {
+    description = "pytest plugin for repeating tests";


Capitalize: https://nixos.org/nixpkgs/manual/#sec-standard-meta-attributes

dotlambda · 2018-08-08T20:27:37Z

pkgs/development/python-modules/distributed/default.nix

+  # ignore 5 cli tests out of 1000 total tests that fail due to subprocesses
+  # these tests are not critical to the library (only the cli)
+  checkPhase = ''
+    py.test distributed -m "not avoid-travis" -r s --timeout-method=thread --timeout=300 --durations=20 --ignore="distributed/cli/tests"


Do you know if there's a way to set the timeout to infinity?

Not sure. I have been following the test instructions that is used on travis.

https://github.com/dask/distributed/blob/master/continuous_integration/travis/run_tests.sh#L1

From https://bitbucket.org/pytest-dev/pytest-timeout/:

Setting a timeout to 0 seconds disables the timeout, so if you have a global timeout set you can still disable the timeout by using the mark.

However, I'd prefer to just get rid of pytest-timeout if that's possible. Otherwise, just set --timeout=0.

Btw you should also try leaving out the other options if possible, especially --durations sounds like something we'd want to avoid. Hydra is usually under heavy load, so tests will run slower than usual.

dotlambda · 2018-08-08T20:28:27Z

@GrahamcOfBorg build python2.pkgs.distributed python3.pkgs.distributed

dotlambda · 2018-08-08T20:29:07Z

@costrouc These commits all look very nice. Thank you for your thorough work!

GrahamcOfBorg · 2018-08-08T20:32:02Z

Failure on x86_64-linux (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed

Partial log (click to expand)

cannot build derivation '/nix/store/clzis8sxxbjxrgcpzj59jd1zpqpwdmjs-python3.6-joblib-0.12.2.drv': 1 dependencies couldn't be built
test_pytest_timeout.py ...................................               [100%]

========================== 35 passed in 94.51 seconds ==========================
cannot build derivation '/nix/store/ixjalzamq475gcf8nkm6lqwymaijfdb0-python2.7-distributed-1.22.1.drv': 3 dependencies couldn't be built
test_pytest_timeout.py ...................................               [100%]

========================= 35 passed in 121.42 seconds ==========================
cannot build derivation '/nix/store/6vbqnpivvxj2vjx2ylq633i6lkpb79hn-python3.6-distributed-1.22.1.drv': 3 dependencies couldn't be built
error: build of '/nix/store/6vbqnpivvxj2vjx2ylq633i6lkpb79hn-python3.6-distributed-1.22.1.drv', '/nix/store/ixjalzamq475gcf8nkm6lqwymaijfdb0-python2.7-distributed-1.22.1.drv' failed

dotlambda · 2018-08-08T20:38:26Z

pkgs/development/python-modules/python-lz4/default.nix

+    sha256 = "1vjfplj37jcw1mf8l810dv76dx0raia3ylgyfy7sfsb3g17brjq6";
+  };
+
+  buildInputs = [ setuptools_scm pkgconfig ];


I think pytestrunner is missing here.

Maybe you didn't encounter distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner') because you have sandboxing turned off? You can turn it on with nix build --option sandbox true I think.

Thank you for showing --option sandbox true yes I can reproduce the issue.

GrahamcOfBorg · 2018-08-08T20:39:22Z

Failure on aarch64-linux (full log)

Attempted: python2.pkgs.pytest-timeout, python3.pkgs.pytest-timeout, python2.pkgs.pytest-faulthandler, python3.pkgs.pytest-faulthandler

Partial log (click to expand)

plugins: timeout-1.3.1
collected 35 items

test_pytest_timeout.py ...................................               [100%]

========================= 35 passed in 517.85 seconds ==========================
test_pytest_timeout.py ...................................               [100%]

========================= 35 passed in 527.26 seconds ==========================
error: build of '/nix/store/n77309chczgsd6bbvkl6cjj4q5v6jw2q-python3.6-pytest-faulthandler-1.5.0.drv', '/nix/store/x1sdwl4z2mfhl301ha5ifnrr89if7cj8-python2.7-pytest-faulthandler-1.5.0.drv' failed

dotlambda · 2018-08-08T20:40:13Z

pkgs/development/python-modules/python-lz4/default.nix

+
+  checkPhase = ''
+    py.test
+  '';


If you add pytestrunner above, this can be removed.

ryantm

Wow, looks like this was a lot of hard work! Please fix the merge conflict.

costrouc · 2018-09-11T03:11:27Z

@ryantm ready for merge. Thanks for the reminder!

ryantm · 2018-09-11T03:20:05Z

@GrahamcOfBorg build python2.pkgs.distributed python3.pkgs.distributed

GrahamcOfBorg · 2018-09-11T03:20:13Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: python2.pkgs.distributed, python3.pkgs.distributed

Partial log (click to expand)

syntax error, unexpected '=', expecting $end, at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/pkgs/development/python-modules/joblib/default.nix:38:9

Cannot nix-instantiate `python3.pkgs.distributed' because:
error: while evaluating the attribute 'buildInputs' of the derivation 'python3.6-distributed-1.22.1' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'callPackageWith' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/customisation.nix:108:35, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/pkgs/top-level/python-packages.nix:4131:12:
while evaluating 'makeOverridable' at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/customisation.nix:67:24, called from /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/lib/customisation.nix:112:8:
syntax error, unexpected '=', expecting $end, at /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/rbvermaa-spot/pkgs/development/python-modules/joblib/default.nix:38:9

GrahamcOfBorg · 2018-09-11T03:20:32Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: python2.pkgs.distributed, python3.pkgs.distributed

Partial log (click to expand)

syntax error, unexpected '=', expecting $end, at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/pkgs/development/python-modules/joblib/default.nix:38:9

Cannot nix-instantiate `python3.pkgs.distributed' because:
error: while evaluating the attribute 'buildInputs' of the derivation 'python3.6-distributed-1.22.1' at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'callPackageWith' at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/lib/customisation.nix:108:35, called from /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/pkgs/top-level/python-packages.nix:4131:12:
while evaluating 'makeOverridable' at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/lib/customisation.nix:67:24, called from /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/lib/customisation.nix:112:8:
syntax error, unexpected '=', expecting $end, at /var/lib/gc-of-borg/nix-test-rs-20/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-20/pkgs/development/python-modules/joblib/default.nix:38:9

GrahamcOfBorg · 2018-09-11T03:20:37Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: python2.pkgs.distributed, python3.pkgs.distributed

Partial log (click to expand)

syntax error, unexpected '=', expecting $end, at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/python-modules/joblib/default.nix:38:9

Cannot nix-instantiate `python3.pkgs.distributed' because:
error: while evaluating the attribute 'buildInputs' of the derivation 'python3.6-distributed-1.22.1' at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/top-level/python-packages.nix:4131:12:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/customisation.nix:112:8:
syntax error, unexpected '=', expecting $end, at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/python-modules/joblib/default.nix:38:9

ryantm · 2018-09-11T03:21:51Z

pkgs/development/python-modules/joblib/default.nix

+    license = lib.licenses.bsd3;
+    maintainers = with lib.maintainers; [ costrouc ];
+  };
+}


This bracket is extra.

- pytest-timeout: no longer requires patch - joblib: was improperly merged with the github merge tool (never using again)

costrouc · 2018-09-11T18:38:34Z

@ryantm I will never use the github merge editor for a quick fix again. That's why the builds failed. I have fixed the issue and tested that the builds succeed now for both joblib and distributed.

ryantm · 2018-09-11T20:05:29Z

@GrahamcOfBorg build python2.pkgs.distributed python3.pkgs.distributed python2.pkgs.joblib python3.pkgs.joblib

GrahamcOfBorg · 2018-09-11T20:18:45Z

Failure on x86_64-darwin (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed, python2.pkgs.joblib, python3.pkgs.joblib

Partial log (click to expand)

3.04s call     distributed/deploy/tests/test_adaptive.py::test_worker_keys
3.00s call     distributed/deploy/tests/test_local.py::test_silent_startup
2.98s call     distributed/tests/test_client_executor.py::test_cancellation
2.63s call     distributed/tests/test_profile.py::test_basic
2.51s call     distributed/tests/test_failed_workers.py::test_restart_sync_no_center
2.37s call     distributed/tests/test_failed_workers.py::test_restart_sync
2.36s call     distributed/diagnostics/tests/test_progress.py::test_AllProgress
== 1 failed, 1051 passed, 114 skipped, 5 xfailed, 4 xpassed in 434.46 seconds ==
builder for '/nix/store/84m9s23yl37v3kwl8xzcks03x74sc6ix-python3.6-distributed-1.22.1.drv' failed with exit code 1
error: build of '/nix/store/84m9s23yl37v3kwl8xzcks03x74sc6ix-python3.6-distributed-1.22.1.drv', '/nix/store/fmmqhlivdvpqzif035d70awqqilqnfm1-python2.7-distributed-1.22.1.drv', '/nix/store/iwxnw0vy2g76f9h1qrvkfcyrwpsh60mf-python2.7-joblib-0.12.4.drv' failed

GrahamcOfBorg · 2018-09-11T20:33:00Z

Failure on x86_64-linux (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed, python2.pkgs.joblib, python3.pkgs.joblib

Partial log (click to expand)

distributed/tests/test_as_completed.py ..........                        [ 21%]
distributed/tests/test_asyncio.py ................s..                    [ 22%]
distributed/tests/test_asyncprocess.py ..........                        [ 23%]
distributed/tests/test_batched.py .......s.s..                           [ 24%]
distributed/tests/test_client.py .......................s.........ss.... [ 28%]
........................sss........X.ss....s.....s.......s.............. [ 34%]
................ssss.............................................s...... [ 40%]
............................/nix/store/i6vl5lwlz5jbkg4r6p340dwmj6fha3xq-stdenv-linux/setup: line 1290:  3123 Aborted                 py.test distributed -m "not avoid-travis" -r s --timeout-method=thread --timeout=0 --durations=20 --ignore="distributed/cli/tests"
builder for '/nix/store/h486hzbaqscqa8acbm0qaryjknlf073d-python3.6-distributed-1.22.1.drv' failed with exit code 134
error: build of '/nix/store/h486hzbaqscqa8acbm0qaryjknlf073d-python3.6-distributed-1.22.1.drv' failed

GrahamcOfBorg · 2018-09-11T21:07:14Z

Timed out, unknown build status on aarch64-linux (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed, python2.pkgs.joblib, python3.pkgs.joblib

Partial log (click to expand)

../../nix/store/8j80sfr6rmn7yd161v1f0f1zqm28b1g6-python3.6-pandas-0.23.4/lib/python3.6/site-packages/pandas/tests/test_strings.py . [  4%]
building of '/nix/store/b5j1ydbjjxkn2nvnxsyc4rsllrzlbkm5-python2.7-pandas-0.23.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/ygqffarb34clqzghzfilrzjniv6viw95-python2.7-partd-0.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/6vr0l3c25khg7s4wj5lwpm4n2jan7sxc-python2.7-dask-0.18.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/kf7c7bsaxzckjf14m7k57h86vfi0flgl-python2.7-distributed-1.22.1.drv': 2 dependencies couldn't be built
building of '/nix/store/qa3qf1ckgx2aiq3ngqm1ir2dqqf3wzp1-python3.6-pandas-0.23.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/qjrb7bqk05ylddl5kz7qzalhl5jmlz3y-python3.6-partd-0.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/niacdp5h3hpc0zclqcz7mcqj8wlmamls-python3.6-dask-0.18.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/9gaq1bb2x16nfrpfc37lc3542zr1kca7-python3.6-distributed-1.22.1.drv': 2 dependencies couldn't be built
error: build of '/nix/store/9gaq1bb2x16nfrpfc37lc3542zr1kca7-python3.6-distributed-1.22.1.drv', '/nix/store/9w6xk081d3r5cs8gkchn4gc56f94nfga-python3.6-joblib-0.12.4.drv', '/nix/store/gws5ach6mbyyjkh44pqssssrf8wfwsd4-python2.7-joblib-0.12.4.drv', '/nix/store/kf7c7bsaxzckjf14m7k57h86vfi0flgl-python2.7-distributed-1.22.1.drv' failed

costrouc · 2018-09-11T21:17:27Z

I have been trying to reproduce the failed tests and everytime I ignore a test another magically seems to fail. I think that turning off the tests is fine. There are 800 tests and only 1 or 2 seem to fail each time. @ryantm is this acceptable? I am sorry that this has taken so long.

ryantm · 2018-09-11T21:55:41Z

@costrouc There are a lot of python packages with the tests disabled. Obviously it is preferred to have them enabled, but if they can't be made reliable, it is better to not have them. Hopefully the package maintainer runs their own tests as part of the release process.

ryantm

please fix test reliability

costrouc · 2018-09-11T22:17:30Z

Fixed. I set doCheck = false for pythonPackages.distributed

ryantm · 2018-09-11T22:28:24Z

@GrahamcOfBorg build python2.pkgs.distributed python3.pkgs.distributed python2.pkgs.joblib python3.pkgs.joblib

GrahamcOfBorg · 2018-09-11T22:38:15Z

Success on x86_64-linux (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed, python2.pkgs.joblib, python3.pkgs.joblib

Partial log (click to expand)

[DEBUG:MainProcess:QueueManagerThread] closing call_queue
[DEBUG:MainProcess:QueueManagerThread] telling queue thread to quit
[DEBUG:MainProcess:QueueManagerThread] joining processes
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
[DEBUG:MainProcess:QueueManagerThread] queue management thread clean shutdown of worker processes: []
[DEBUG:MainProcess:MainThread] running the remaining "atexit" finalizers
/nix/store/2q33qrmnphzc7hr4pdams7cb8z6sbqm7-python2.7-distributed-1.22.1
/nix/store/93ipmi0zapd8ckfxrrgrqsld7jdbl62i-python3.6-distributed-1.22.1
/nix/store/24i4w686088i6k6pmxkrd2k8cnfgiil9-python2.7-joblib-0.12.4
/nix/store/13ldlwmd3qj4ihhkn0cp2g0agb1klxkc-python3.6-joblib-0.12.4

ryantm · 2018-09-11T23:06:17Z

Woo! 🎉

GrahamcOfBorg · 2018-09-11T23:14:25Z

Failure on x86_64-darwin (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed, python2.pkgs.joblib, python3.pkgs.joblib

Partial log (click to expand)

[DEBUG:MainProcess:Dummy-677] Interpreter shutting down. Waking up queue_manager_threads [(<Thread(QueueManagerThread, started daemon 123145423290368)>, <joblib.externals.loky.process_executor._ThreadWakeup instance at 0x10b84c4d0>)]
[DEBUG:MainProcess:QueueManagerThread] queue management thread shutting down
[DEBUG:MainProcess:QueueManagerThread] closing call_queue
[DEBUG:MainProcess:QueueManagerThread] telling queue thread to quit
[DEBUG:MainProcess:QueueManagerThread] joining processes
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
[DEBUG:MainProcess:QueueManagerThread] queue management thread clean shutdown of worker processes: []
[DEBUG:MainProcess:Dummy-677] running the remaining "atexit" finalizers
builder for '/nix/store/iwxnw0vy2g76f9h1qrvkfcyrwpsh60mf-python2.7-joblib-0.12.4.drv' failed with exit code 1
error: build of '/nix/store/iwxnw0vy2g76f9h1qrvkfcyrwpsh60mf-python2.7-joblib-0.12.4.drv' failed

GrahamcOfBorg · 2018-09-11T23:29:35Z

Timed out, unknown build status on aarch64-linux (full log)

Attempted: python2.pkgs.distributed, python3.pkgs.distributed, python2.pkgs.joblib, python3.pkgs.joblib

Partial log (click to expand)

building of '/nix/store/9w6xk081d3r5cs8gkchn4gc56f94nfga-python3.6-joblib-0.12.4.drv' timed out after 3600 seconds
building of '/nix/store/b5j1ydbjjxkn2nvnxsyc4rsllrzlbkm5-python2.7-pandas-0.23.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/ygqffarb34clqzghzfilrzjniv6viw95-python2.7-partd-0.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/6vr0l3c25khg7s4wj5lwpm4n2jan7sxc-python2.7-dask-0.18.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/bfc1k8ncmfqij286lkm75zi63vv4zdyw-python2.7-distributed-1.22.1.drv': 1 dependencies couldn't be built
building of '/nix/store/qa3qf1ckgx2aiq3ngqm1ir2dqqf3wzp1-python3.6-pandas-0.23.4.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/qjrb7bqk05ylddl5kz7qzalhl5jmlz3y-python3.6-partd-0.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/niacdp5h3hpc0zclqcz7mcqj8wlmamls-python3.6-dask-0.18.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/fv7f8y18rhl0zcy0zqmacbrv5aawmb0a-python3.6-distributed-1.22.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/9w6xk081d3r5cs8gkchn4gc56f94nfga-python3.6-joblib-0.12.4.drv', '/nix/store/bfc1k8ncmfqij286lkm75zi63vv4zdyw-python2.7-distributed-1.22.1.drv', '/nix/store/fv7f8y18rhl0zcy0zqmacbrv5aawmb0a-python3.6-distributed-1.22.1.drv', '/nix/store/gws5ach6mbyyjkh44pqssssrf8wfwsd4-python2.7-joblib-0.12.4.drv' failed

ryantm · 2018-09-11T23:36:41Z

@costrouc It looks like the tests of joblib are failing on darwin :(

costrouc requested a review from FRidh as a code owner August 8, 2018 02:27

GrahamcOfBorg added the 6.topic: python label Aug 8, 2018

costrouc mentioned this pull request Aug 8, 2018

New dask-jobqueue 0.3.0, docrep 0.2.3. Updated distributed 1.15.1 -> 1.22.1 #44637

Closed

9 tasks

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Aug 8, 2018

dotlambda reviewed Aug 8, 2018

View reviewed changes

costrouc force-pushed the costrouc/dask-distributed branch from 2211599 to 44e44ed Compare August 8, 2018 19:40

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 and removed 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Aug 8, 2018

dotlambda reviewed Aug 8, 2018

View reviewed changes

GrahamcOfBorg added 2.status: merge conflict and removed 2.status: merge conflict labels Sep 6, 2018

ryantm requested changes Sep 11, 2018

View reviewed changes

merge: fixing changes with nixpkgs since pull-request

dac171d

- pytest-timeout: no longer requires patch - joblib: was improperly merged with the github merge tool (never using again)

costrouc force-pushed the costrouc/dask-distributed branch from 5b9dfe3 to dac171d Compare September 11, 2018 18:26

ryantm approved these changes Sep 11, 2018

View reviewed changes

fixup! merge: fixing changes with nixpkgs since pull-request

027b9be

ryantm requested changes Sep 11, 2018

View reviewed changes

ryantm approved these changes Sep 11, 2018

View reviewed changes

ryantm merged commit 8d4248a into NixOS:master Sep 11, 2018

pythonPackages.distributed: 1.15.1 -> 1.22.1 #44657

pythonPackages.distributed: 1.15.1 -> 1.22.1 #44657

Conversation

costrouc commented Aug 8, 2018 • edited

Motivation for this change

Things done

costrouc commented Aug 8, 2018

xeji commented Aug 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costrouc commented Aug 8, 2018 • edited

dotlambda commented Aug 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costrouc Aug 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Aug 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Aug 8, 2018

dotlambda commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Aug 8, 2018

Choose a reason for hiding this comment

ryantm left a comment

Choose a reason for hiding this comment

costrouc commented Sep 11, 2018

ryantm commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

Choose a reason for hiding this comment

costrouc commented Sep 11, 2018 • edited

ryantm commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

costrouc commented Sep 11, 2018

ryantm commented Sep 11, 2018

ryantm left a comment

Choose a reason for hiding this comment

costrouc commented Sep 11, 2018

ryantm commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

ryantm commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

ryantm commented Sep 11, 2018

costrouc commented Aug 8, 2018 •

edited

costrouc commented Aug 8, 2018 •

edited

dotlambda commented Aug 8, 2018 •

edited

costrouc Aug 8, 2018 •

edited

costrouc commented Sep 11, 2018 •

edited