pythonPackages.pykdtree: init at 1.3.0 #41355

PsyanticY · 2018-06-01T12:24:06Z

Motivation for this change

pykdtree is a kd-tree implementation for fast nearest neighbour search in Python. The aim is to be the fastest implementation around for common use cases (low dimensions and low number of neighbours) for both tree construction and queries.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

FRidh · 2018-06-02T10:50:43Z

@GrahamcOfBorg build pythonPackages.pykdtree python3Packages.pykdtree

GrahamcOfBorg · 2018-06-02T10:51:13Z

Failure on x86_64-darwin (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree/kdtree.c:529:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 warning and 1 error generated.
1 warning and 1 error generated.
error: command 'clang' failed with exit status 1
error: command 'clang' failed with exit status 1
builder for '/nix/store/q8ddc3mb723lhfhcrymk2m3pxascziif-python2.7-pykdtree-1.3.0.drv' failed with exit code 1
builder for '/nix/store/hyifzfa4q7jrb2881mgsab4ad34592q4-python3.6-pykdtree-1.3.0.drv' failed with exit code 1
error: build of '/nix/store/hyifzfa4q7jrb2881mgsab4ad34592q4-python3.6-pykdtree-1.3.0.drv', '/nix/store/q8ddc3mb723lhfhcrymk2m3pxascziif-python2.7-pykdtree-1.3.0.drv' failed

GrahamcOfBorg · 2018-06-02T10:51:37Z

Success on x86_64-linux (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.020s

OK
/nix/store/yl94izcvc6a7i5l72fg9abb40glk3xpp-python2.7-pykdtree-1.3.0
/nix/store/fh6ciy1w81vfjij1flkjp99via4ic324-python3.6-pykdtree-1.3.0

GrahamcOfBorg · 2018-06-02T10:51:50Z

Success on aarch64-linux (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.084s

OK
/nix/store/izny5w0wwpffbh90xsh0gy5dan56xlsn-python2.7-pykdtree-1.3.0
/nix/store/1k0n3dpwcss755p8p1k221spnhhn2w0p-python3.6-pykdtree-1.3.0

xeji · 2018-06-03T20:23:56Z

looks like this needs openmp as an extra build input, at least on Darwin.

PsyanticY · 2018-06-05T11:35:25Z

@xeji Thanks for the review. According to the package documentation we can disable the use of openmp by exporting USE_OMP=0. I was not able to add openmp as build input as it is not under all-packages.nix.

xeji · 2018-06-05T12:00:30Z

Apparently openmp is in llvmPackages.openmp, so you can add it like this (I didn't check if it really works, just copied from another package):

pykdtree = callPackage ../development/python-modules/pykdtree { 
  inherit (llvmPackages) openmp;
};

But disabling it should also be fine, so let's try that first.

@GrahamcOfBorg build pythonPackages.pykdtree python3Packages.pykdtree

GrahamcOfBorg · 2018-06-05T12:01:16Z

Success on x86_64-linux (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.011s

OK
/nix/store/c2dkpjykh3nk6dsiimgsy15pm86dwwpd-python2.7-pykdtree-1.3.0
/nix/store/kc5v6dyfwrws9pvy21a5qmpc687757hg-python3.6-pykdtree-1.3.0

GrahamcOfBorg · 2018-06-05T12:01:17Z

Success on aarch64-linux (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.074s

OK
/nix/store/kazgrm1s9gkd184r17dwhh52mkc4g84h-python2.7-pykdtree-1.3.0
/nix/store/9fpaj3ic7kqfm1dbj2xw3kcwp316sqb9-python3.6-pykdtree-1.3.0

…Input)

PsyanticY · 2018-06-05T13:17:11Z

@xeji Seems like no build was triggered for darwin.

xeji · 2018-06-05T15:23:48Z

@Assassinkin it was queued but currently no darwin builder is available (they're run by volunteers and not always up). See https://monitoring.nix.ci/dashboard/db/ofborg?refresh=10s&orgId=1 for status.
So unfortunately we need to wait.

PsyanticY · 2018-06-05T15:39:42Z

@xeji Thanks for the explanation didn't know about that. In the mean time i was able to add openmp as build input by passing it as follow

 pykdtree = callPackage ../development/python-modules/pykdtree {
    inherit (pkgs.llvmPackages) openmp;
  };

What do you recommend ? use openmp or not (if the current fix do work)

xeji · 2018-06-05T15:55:18Z

According to upstream https://github.com/storpipfugl/pykdtree#benchmarks , multithreading with openmp has some performance benefits, so I would enable it if it works.

PsyanticY · 2018-06-05T17:54:32Z

@xeji included openmp.

xeji · 2018-06-05T18:15:51Z

@GrahamcOfBorg build pythonPackages.pykdtree python3Packages.pykdtree

GrahamcOfBorg · 2018-06-05T18:16:40Z

Success on aarch64-linux (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.084s

OK
/nix/store/hg4rsrwq4gk9dygrs7l3qa2vyqqbvi25-python2.7-pykdtree-1.3.0
/nix/store/7lv01v1v0cma0xbmvp62y9xkbx7h2pff-python3.6-pykdtree-1.3.0

GrahamcOfBorg · 2018-06-05T18:16:54Z

Success on x86_64-linux (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.014s

OK
/nix/store/pxkgi1vis7asyczw13s1yfmax7z4aayi-python2.7-pykdtree-1.3.0
/nix/store/jdbjwzwnyjbhnr6j5lx7vli5a2dagdjb-python3.6-pykdtree-1.3.0

xeji · 2018-06-05T18:19:31Z

Thanks. Let's wait for the Darwin build, I'll merge if it succeeds.

GrahamcOfBorg · 2018-06-05T19:37:44Z

Failure on x86_64-darwin (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

/nix/store/jsildbay8mxa92jiq93pm06awh8f7qaw-python3.6-numpy-1.14.3/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " \
 ^
pykdtree/kdtree.c:529:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 warning and 1 error generated.
error: command 'clang' failed with exit status 1
builder for '/nix/store/6jv5hklbng0v7dja8bxcn74m77qi4ijx-python3.6-pykdtree-1.3.0.drv' failed with exit code 1
error: build of '/nix/store/6jv5hklbng0v7dja8bxcn74m77qi4ijx-python3.6-pykdtree-1.3.0.drv', '/nix/store/6rpxih7vwvm0gh3n7h4r5snxz0535mg1-python2.7-pykdtree-1.3.0.drv' failed

xeji · 2018-06-05T19:40:24Z

that was the old build job, we can ignore it.

GrahamcOfBorg · 2018-06-05T22:23:33Z

Success on x86_64-darwin (full log)

Attempted: pythonPackages.pykdtree, python3Packages.pykdtree

Partial log (click to expand)

pykdtree.test_tree.test1d_mask ... ok
pykdtree.test_tree.test1d_all_masked ... ok
pykdtree.test_tree.test3d_mask ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.016s

OK
/nix/store/01ypzzi5cfya26xf7q50hbxz9rvn48c7-python2.7-pykdtree-1.3.0
/nix/store/lrd33j9lg7986ana0yjcdj06ils7yj7p-python3.6-pykdtree-1.3.0

* master: (63 commits) tests.cc-wrapper: do not test sanitizers on darwin nodePackages_8_x: Add nodejs (npm) to wrapper python.pkgs.cairocffi: support darwin osinfo-db: 20180514 -> 20180531 (#41532) pkcs11helper: 1.22 -> 1.23 (#41530) pmd: 6.3.0 -> 6.4.0 (#41529) renpy: 6.99.14.3 -> 7.0.0 (#41515) wireless-regdb: 2018.05.09 -> 2018.05.31 (#41502) gnuplot: 5.2.3 -> 5.2.4 (#41505) qrencode: 4.0.0 -> 4.0.1 Revert "libbsd: replace with nbcompat" Revert "opendkim: fix libbsd dependency" Revert "libbsd-freedesktop: re-init at 0.9.1, use for samba" flent: fix fetchFromGitHub version syncthing: 0.14.47 -> 0.14.48 pythonPackages.pykdtree: init at 1.3.0 (#41355) snd: 18.3 -> 18.4 bat: support darwin faust2: fix build with llvm 5.0.2 (#40672) mc: 4.8.20 -> 4.8.21 (#41496) ...

pythonPackages.pykdtree: init at 1.3.0

4d45599

PsyanticY requested a review from FRidh as a code owner June 1, 2018 12:24

GrahamcOfBorg added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jun 1, 2018

pythonPackages.pykdtree fix build on darwin

6dee806

pythonPackages.pykdtree: fix build on darwin (include openmp as build…

cb27802

…Input)

xeji merged commit 26c194d into NixOS:master Jun 5, 2018

PsyanticY deleted the pykdtree branch June 11, 2018 13:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.pykdtree: init at 1.3.0 #41355

pythonPackages.pykdtree: init at 1.3.0 #41355

PsyanticY commented Jun 1, 2018

FRidh commented Jun 2, 2018

GrahamcOfBorg commented Jun 2, 2018

GrahamcOfBorg commented Jun 2, 2018

GrahamcOfBorg commented Jun 2, 2018

xeji commented Jun 3, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

pythonPackages.pykdtree: init at 1.3.0 #41355

pythonPackages.pykdtree: init at 1.3.0 #41355

Conversation

PsyanticY commented Jun 1, 2018

Motivation for this change

Things done

FRidh commented Jun 2, 2018

GrahamcOfBorg commented Jun 2, 2018

GrahamcOfBorg commented Jun 2, 2018

GrahamcOfBorg commented Jun 2, 2018

xeji commented Jun 3, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

PsyanticY commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018

xeji commented Jun 5, 2018

GrahamcOfBorg commented Jun 5, 2018