pythonPackages.hdbscan: init at 0.8.12 #37762

ixxie · 2018-03-25T12:41:20Z

Initial packaging of the python HDBSCAN module and the related variant HDBSCAN with cosine distance.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

ixxie · 2018-03-27T07:46:38Z

@dotlambda - got time to review this?

dotlambda

Can you explain the difference between the with-cosine-distance version and the one without? What's the difference in code? http://github.com/scikit-learn-contrib/hdbscan does not seem to be the upstream repo of hdbscan-with-cosine-distance.

dotlambda · 2018-03-30T15:42:12Z

pkgs/development/python-modules/hdbscan-with-cosine-distance/default.nix

+  propagatedBuildInputs = [ cython numpy scipy scikitlearn ];
+
+  meta = with lib; {
+    description = "HDBSCAN - Hierarchical Density-Based Spatial Clustering of Applications with Noise. HDBSCAN is a clustering algorithm with a scikit-learn compatible API; it doesn't require defining the number of clusters sought, and fares well with oddly shaped clusters.";


This is to long, might be used as longDescription. Please have a look at https://nixos.org/nixpkgs/manual/#sec-standard-meta-attributes.

dotlambda · 2018-03-30T15:42:40Z

pkgs/development/python-modules/hdbscan/default.nix

+  propagatedBuildInputs = [ cython numpy scipy scikitlearn ];
+
+  meta = with lib; {
+    description = "HDBSCAN - Hierarchical Density-Based Spatial Clustering of Applications with Noise. HDBSCAN is a clustering algorithm with a scikit-learn compatible API; it doesn't require defining the number of clusters sought, and fares well with oddly shaped clusters.";


Too long as well.

Should have learned this by now, but its somehow counter-intuitive; fixed now.

dotlambda · 2018-03-30T15:48:13Z

pkgs/development/python-modules/hdbscan/default.nix

+  };
+
+  # No tests in archive
+  doCheck = false;


Can we fetch from GitHub instead and use something like nostetests -s hdbscan as the checkPhase?

Done, but checks are in installCheck instead.

ixxie · 2018-04-01T19:24:53Z

@dotlambda - I asked around and it turns out hdbscan-with-cosine-distance is deprecated since its merged back with the main package; I removed it from this PR.

dotlambda · 2018-04-03T09:10:34Z

pkgs/development/python-modules/hdbscan/default.nix

+  src = fetchFromGitHub {
+    owner = "scikit-learn-contrib";
+    repo = "hdbscan";
+    rev = "51fd5ed225d971daa7658d94814c8237a75a03e4";


That's not the revision for tag 0.8.12.

Hmm I have always been taking the latest revision instead of the tags; I'll keep an eye for it now. Should be fixed now.

dotlambda · 2018-04-04T19:40:06Z

pkgs/development/python-modules/hdbscan/default.nix

+  src = fetchFromGitHub {
+    owner = "scikit-learn-contrib";
+    repo = "hdbscan";
+    rev = "12c301b03ca82f66e09a1e7ef9a8b6dd9265b1ce";


You should use rev = version;.

dotlambda · 2018-04-04T19:40:44Z

@GrahamcOfBorg python2.pkgs.hdbscan python3.pkgs.hdbscan

ixxie · 2018-04-06T10:49:22Z

@dotlambda - should be okay now, unless there are other issues.

dotlambda · 2018-04-06T10:52:46Z

@GrahamcOfBorg build python2.pkgs.hdbscan python3.pkgs.hdbscan

GrahamcOfBorg · 2018-04-06T10:53:32Z

Failure on x86_64-linux (full log)

Attempted: python2.pkgs.hdbscan, python3.pkgs.hdbscan

Partial log (click to expand)

trying https://github.com/scikit-learn-contrib/hdbscan/archive/0.8.12.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   136    0   136    0     0    247      0 --:--:-- --:--:-- --:--:--   247
100 10.0M    0 10.0M    0     0  2146k      0 --:--:--  0:00:04 --:--:-- 3092k
unpacking source archive /build/0.8.12.tar.gz
fixed-output derivation produced path '/nix/store/kklhm9yrxg9z3cryjp8y90hdv98hsphj-source' with sha256 hash '1z8396z0fh7w6gv2hgb7cykn9qm7l4pzgvk9kbdi18mvba6bq9fi' instead of the expected hash '1rs4kwd5nbb1x1n29h7saygynqyyyi7mqmra3fxc9hdc3flazvcn'
cannot build derivation '/nix/store/086wvl5rljfzn4byiksl5j78x4b6lvk2-python2.7-hdbscan-0.8.12.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/qcgkymriykxisaa5fmhz3f0hjyflw3ai-python3.6-hdbscan-0.8.12.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/086wvl5rljfzn4byiksl5j78x4b6lvk2-python2.7-hdbscan-0.8.12.drv', '/nix/store/qcgkymriykxisaa5fmhz3f0hjyflw3ai-python3.6-hdbscan-0.8.12.drv' failed

GrahamcOfBorg · 2018-04-06T11:30:21Z

Failure on aarch64-linux (full log)

Attempted: python2.pkgs.hdbscan, python3.pkgs.hdbscan

Partial log (click to expand)

>>  raise AssertionError('\nArrays are not almost equal to 15 decimals\n\n(mismatch 0.793650793651%)\n x: array([[         0,          0,          2,          0,          0,\n                 0,          0,          0,          0,          0,\n                -5,          0,          0,          0,          0,...\n y: array([[         0,          0,          2,          0,          0,\n                 0,          0,          0,          0,          0,\n                -5,          0,          0,          0,          0,...')


----------------------------------------------------------------------
Ran 8229 tests in 1477.230s

FAILED (SKIP=64, failures=1)
builder for '/nix/store/19nzi7ikfhgz0csk3y8hbhj8wpxh55jy-python2.7-scikit-learn-0.19.1.drv' failed with exit code 1
cannot build derivation '/nix/store/ariqzmiq9i6icsmmyq64c54clny8cw78-python2.7-hdbscan-0.8.12.drv': 2 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/ariqzmiq9i6icsmmyq64c54clny8cw78-python2.7-hdbscan-0.8.12.drv', '/nix/store/jj528nrarx23i5h5zvrpibklxpfzdxl3-python3.6-hdbscan-0.8.12.drv' failed

dotlambda · 2018-04-08T07:38:39Z

The hash seems to be wrong.

ixxie · 2018-04-08T08:37:27Z

@dotlambda well I did think it was weird when I replaced the rev with version and it built without need to alter the hash; I guess there was a caching issue? Anyway I will check it out.

ixxie · 2018-04-25T18:49:25Z

@dotlambda numpy fixes are in so I reverted to fetchPypi and enabled the tests. I think its ready now.

dotlambda · 2018-04-26T06:09:42Z

@GrahamcOfBorg build python2.pkgs.hdbscan python3.pkgs.hdbscan

dotlambda · 2018-04-26T06:56:33Z

pkgs/development/python-modules/hdbscan/default.nix

+
+  checkInputs = [ nose ];
+
+  installCheck = ''


This should be checkPhase (which is actually installCheckPhase for python packages).

GrahamcOfBorg · 2018-04-26T07:10:20Z

Failure on aarch64-linux (full log)

Attempted: python2.pkgs.hdbscan, python3.pkgs.hdbscan

Partial log (click to expand)

                 from /nix/store/savp4vc8d7i9nmb7wcsdx3prakp37vgv-python2.7-numpy-1.14.2/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from extension_type_memoryview.cpp:580:
/nix/store/savp4vc8d7i9nmb7wcsdx3prakp37vgv-python2.7-numpy-1.14.2/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
building of '/nix/store/g9ccyb85m5kxbfb3gy26840d90qxsf9k-python2.7-Cython-0.28.1.drv' timed out after 3600 seconds
building of '/nix/store/l60jan1mgs2yd0lnkmsprzjapx4rynv9-python3.6-Cython-0.28.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/zn3mx7hvjjslbw2jzz4nw2pr708dzq89-python2.7-hdbscan-0.8.12.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/qf2g8l733swq4bm3yjmzz9aml0b2gl5c-python3.6-hdbscan-0.8.12.drv': 2 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/qf2g8l733swq4bm3yjmzz9aml0b2gl5c-python3.6-hdbscan-0.8.12.drv', '/nix/store/zn3mx7hvjjslbw2jzz4nw2pr708dzq89-python2.7-hdbscan-0.8.12.drv' failed

GrahamcOfBorg · 2018-04-26T18:48:57Z

Success on x86_64-linux (full log)

Attempted: python2.pkgs.hdbscan, python3.pkgs.hdbscan

Partial log (click to expand)

hdbscan.tests.test_rsl.test_rsl_high_dimensional ... ok
hdbscan.tests.test_rsl.test_rsl_badargs ... ok
hdbscan.tests.test_rsl.test_rsl_is_sklearn_estimator ... ok

----------------------------------------------------------------------
Ran 40 tests in 8.006s

OK (skipped=5)
/nix/store/0y8icid2x4nh0imqrgpgf4bwj7kh24kc-python2.7-hdbscan-0.8.12
/nix/store/l473knp8nym3ywhi2zyrqbgcx60zmsmz-python3.6-hdbscan-0.8.12

ixxie · 2018-04-28T12:58:54Z

@dotlambda - using installCheckPhase didn't work for that particular call but removing an explicit check script call made the tests function correctly as far as I can tell.

dotlambda · 2018-04-29T17:53:33Z

@GrahamcOfBorg build python2.pkgs.hdbscan python3.pkgs.hdbscan

GrahamcOfBorg · 2018-04-29T17:55:05Z

Success on x86_64-linux (full log)

Attempted: python2.pkgs.hdbscan, python3.pkgs.hdbscan

Partial log (click to expand)

hdbscan.tests.test_rsl.test_rsl_high_dimensional ... ok
hdbscan.tests.test_rsl.test_rsl_badargs ... ok
hdbscan.tests.test_rsl.test_rsl_is_sklearn_estimator ... ok

----------------------------------------------------------------------
Ran 40 tests in 4.821s

OK (skipped=5)
/nix/store/8lga7r4pclf96v863hnbpqcgaxcb6fzf-python2.7-hdbscan-0.8.12
/nix/store/jhcwhfcrzxvh9dfncanzyjj35xk6s6mm-python3.6-hdbscan-0.8.12

GrahamcOfBorg · 2018-04-29T18:54:07Z

Failure on aarch64-linux (full log)

Attempted: python2.pkgs.hdbscan, python3.pkgs.hdbscan

Partial log (click to expand)

                 from /nix/store/9ib9731m2av0dxpnj36gdwr48c6qmzvd-python2.7-numpy-1.14.2/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from extension_type_memoryview.cpp:580:
/nix/store/9ib9731m2av0dxpnj36gdwr48c6qmzvd-python2.7-numpy-1.14.2/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
building of '/nix/store/kh3qnjl7ik5ap4awv41hx6inl5k390sj-python2.7-Cython-0.28.1.drv' timed out after 3600 seconds
building of '/nix/store/dqfscwhjqfxd73arwp1gxf1y2wh0k6dm-python3.6-Cython-0.28.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/jbqjm20h62zgviqvbgby5f6g3nycbk11-python2.7-hdbscan-0.8.12.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/k3rl7j9034y282c1ix46cyd57rdhvsc4-python3.6-hdbscan-0.8.12.drv': 2 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/jbqjm20h62zgviqvbgby5f6g3nycbk11-python2.7-hdbscan-0.8.12.drv', '/nix/store/k3rl7j9034y282c1ix46cyd57rdhvsc4-python3.6-hdbscan-0.8.12.drv' failed

ixxie requested a review from FRidh as a code owner March 25, 2018 12:41

GrahamcOfBorg added the 6.topic: python label Mar 25, 2018

ixxie changed the title ~~pythonPackage.shdbscan and pythonPackages.hdbscan-with-cosine-distance~~ pythonPackage.hdbscan and pythonPackages.hdbscan-with-cosine-distance Mar 25, 2018

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Mar 25, 2018

ixxie changed the title ~~pythonPackage.hdbscan and pythonPackages.hdbscan-with-cosine-distance~~ pythonPackages.hdbscan and pythonPackages.hdbscan-with-cosine-distance Mar 25, 2018

dotlambda requested changes Mar 30, 2018

View reviewed changes

ixxie changed the title ~~pythonPackages.hdbscan and pythonPackages.hdbscan-with-cosine-distance~~ pythonPackages.hdbscan: init at 0.8.12 Apr 1, 2018

dotlambda reviewed Apr 3, 2018

View reviewed changes

dotlambda reviewed Apr 4, 2018

View reviewed changes

dotlambda reviewed Apr 26, 2018

View reviewed changes

pythonPackages.hdbscan: init at 0.8.12

4b8b513

nitpicks

f716e4a

dotlambda merged commit 80bf73b into NixOS:master Apr 29, 2018

Synthetica9 pushed a commit to Synthetica9/nixpkgs that referenced this pull request May 3, 2018

pythonPackages.hdbscan: init at 0.8.12 (NixOS#37762)

498284d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.hdbscan: init at 0.8.12 #37762

pythonPackages.hdbscan: init at 0.8.12 #37762

ixxie commented Mar 25, 2018

ixxie commented Mar 27, 2018

dotlambda left a comment

dotlambda Mar 30, 2018

dotlambda Mar 30, 2018

ixxie Apr 1, 2018

dotlambda Mar 30, 2018

ixxie Apr 1, 2018

ixxie commented Apr 1, 2018

dotlambda Apr 3, 2018

ixxie Apr 4, 2018

dotlambda Apr 4, 2018

dotlambda commented Apr 4, 2018

ixxie commented Apr 6, 2018

dotlambda commented Apr 6, 2018 •

edited

GrahamcOfBorg commented Apr 6, 2018

GrahamcOfBorg commented Apr 6, 2018

dotlambda commented Apr 8, 2018

ixxie commented Apr 8, 2018

ixxie commented Apr 25, 2018

dotlambda commented Apr 26, 2018

dotlambda Apr 26, 2018

GrahamcOfBorg commented Apr 26, 2018

GrahamcOfBorg commented Apr 26, 2018

ixxie commented Apr 28, 2018

dotlambda commented Apr 29, 2018

GrahamcOfBorg commented Apr 29, 2018

GrahamcOfBorg commented Apr 29, 2018

pythonPackages.hdbscan: init at 0.8.12 #37762

pythonPackages.hdbscan: init at 0.8.12 #37762

Conversation

ixxie commented Mar 25, 2018

Things done

ixxie commented Mar 27, 2018

dotlambda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixxie commented Apr 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Apr 4, 2018

ixxie commented Apr 6, 2018

dotlambda commented Apr 6, 2018 • edited

GrahamcOfBorg commented Apr 6, 2018

GrahamcOfBorg commented Apr 6, 2018

dotlambda commented Apr 8, 2018

ixxie commented Apr 8, 2018

ixxie commented Apr 25, 2018

dotlambda commented Apr 26, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Apr 26, 2018

GrahamcOfBorg commented Apr 26, 2018

ixxie commented Apr 28, 2018

dotlambda commented Apr 29, 2018

GrahamcOfBorg commented Apr 29, 2018

GrahamcOfBorg commented Apr 29, 2018

dotlambda commented Apr 6, 2018 •

edited