pythonPackages.chainer: init at 3.2.0 #33123

hyphon81 · 2017-12-28T04:02:48Z

cupy: init at 2.2.0

nccl: init at 1.3.4

filelock: init at 2.0.13

fastrlock: init at 0.3

Motivation for this change

Adding the chainer and related packages.
The chainer is a python framework for writing neural networks codes.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

jtojnar · 2017-12-28T04:20:20Z

pkgs/development/python-modules/chainer/default.nix

+}:
+
+with pkgs;
+with python.pkgs;


Instead of using python.pkgs, it is preferred to list the python dependencies as arguments to the derivation.

jtojnar · 2017-12-28T04:24:58Z

pkgs/development/libraries/science/math/nccl/default.nix

+  name = "cudatoolkit-${cudatoolkit.majorVersion}-nccl-${version}";
+  version = "1.3.4-1";
+
+  src = fetchurl {


fetchFromGitHub should be used instead https://nixos.org/nixpkgs/manual/#sec-sources

jtojnar · 2017-12-28T04:25:38Z

pkgs/development/libraries/science/math/nccl/default.nix

@@ -0,0 +1,43 @@
+{ stdenv, pkgs, fetchurl, cudatoolkit }:


Dependencies should be listed explicitly, instead of using pkgs.

jtojnar · 2017-12-28T04:28:49Z

pkgs/development/libraries/science/math/nccl/default.nix

+      NVIDIA Collective Communications Library.
+      Multi-GPU and multi-node collective communication primitives.
+    '';
+    homepage = "https://developer.nvidia.com/nccl";


When possible, unquoted URLs are preferred.

jtojnar · 2017-12-28T07:38:42Z

pkgs/development/python-modules/cupy/default.nix

+
+  CUDA_PATH = "${cudatoolkit}";
+  CFLAGS = "-I ${cudnn}/include -I ${nccl}/include";
+  LDFLAGS = "-L ${cudnn}/lib -L ${nccl}/lib";


Are these three lines necessary? It builds fine for me without them.

According to cupy's official install guide document, we need set environment values when cupy will be installed with cuda, cudnn, nccl.
https://docs-cupy.chainer.org/en/stable/install.html#install-cupy-with-cuda
https://docs-cupy.chainer.org/en/stable/install.html#install-cupy-with-cudnn-and-nccl

If I understand the note correctly, it should not be necessary. Nix will add ${cudatoolkit}/bin/nvcc to PATH, as well as set the appropriate CFLAGS and LDFLAGS.

The official document prefers to use pip installing and set envs with 'export'.
I'm worrying those environment values are needed in python phase.
I think Nix will not add CFLAGS and LDFLAGS to python.

Actually, those checks are in python.
https://github.com/cupy/cupy/blob/06e84b561aeba2d4abc5342fe5a7c1da894a436b/install/build.py#L33
https://github.com/cupy/cupy/blob/df93794d685560ac6f44b23528e81eb501e7871c/cupy_setup_build.py#L250

Nix sets those variables in python since it is based on mkDerivation.

Sorry.
You are right, those settings are unnecessary.
I delete those.

I found out CUDA_PATH is unnecessary when PATH has ${cudatoolkit}/bin.
And I attempted building after delete those environment variables settings, and
chainer could find cudnn.

>>> import chainer >>> chainer.cuda.available True >>> chainer.cuda.cudnn_enabled True >>>

jtojnar · 2017-12-28T08:04:01Z

pkgs/development/libraries/science/math/nccl/default.nix

+  src = fetchFromGitHub {
+    owner = "NVIDIA";
+    repo = "nccl";
+    rev = "649f04d07711440ec2699ce7ec1af4ab71d6dbf9";


You can use rev = "v${version}";

jtojnar · 2017-12-28T09:02:11Z

pkgs/development/libraries/science/math/nccl/default.nix

+    cudatoolkit
+  ];
+
+  BUILDDIR = "./";


This should not be needed, the default build/ directory should be fine.

jtojnar · 2017-12-28T09:04:19Z

pkgs/development/libraries/science/math/nccl/default.nix

+  BUILDDIR = "./";
+  PREFIX = "$out";
+  CUDA_HOME = "${cudatoolkit}";
+  CUDA_LIB = "${cudatoolkit.lib}/lib";


These three variables should be in makeFlags:

makeFlags = [ "PREFIX=$(out)" "CUDA_HOME=${cudatoolkit}" "CUDA_LIB=${cudatoolkit.lib}/lib" ];

jtojnar · 2017-12-28T09:07:43Z

pkgs/development/libraries/science/math/nccl/default.nix

+  ];
+
+  BUILDDIR = "./";
+  PREFIX = "$out";


This will set the PREFIX variable to literal $out string. The out variable needs to be interpolated somewhere, either by shell using preInstall = ''export PREFIX="$out"'';, or by make as suggested below. Correctly setting the variable will also allow you to get rid of the postInstall section.

jtojnar · 2017-12-28T10:09:23Z

@GrahamcOfBorg build nccl python2Packages.chainer python2Packages.cupy python3Packages.chainer python3Packages.cupy

GrahamcOfBorg

Failure for system: aarch64-linux

Package ‘cudatoolkit-8.0.61’ in /var/lib/gc-of-borg/nix-test-rs-1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-1/pkgs/development/compilers/cudatoolkit/default.nix:128 has an unfree license (‘unfree’), refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg

Failure for system: x86_64-darwin

Package ‘cudatoolkit-8.0.61’ in /Users/graham/nix-borg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-zoidberg/pkgs/development/compilers/cudatoolkit/default.nix:128 has an unfree license (‘unfree’), refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg

Failure for system: x86_64-linux

Package ‘cudatoolkit-8.0.61’ in /var/lib/gc-of-borg/.nix-test-rs/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-zoidberg/pkgs/development/compilers/cudatoolkit/default.nix:128 has an unfree license (‘unfree’), refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

jtojnar · 2017-12-28T10:11:18Z

LGTM, could you please squash the fix-up commits?

FRidh · 2017-12-28T12:02:26Z

pkgs/development/python-modules/chainer/default.nix

+}:
+
+buildPythonPackage rec {
+  name = "${pname}-${version}";


name can be removed from buildPython* expressions

jtojnar · 2017-12-28T16:53:41Z

Thanks for your work.

hyphon81 requested a review from FRidh as a code owner December 28, 2017 04:02

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Dec 28, 2017

jtojnar requested changes Dec 28, 2017

View reviewed changes

jtojnar added 6.topic: python 8.has: package (new) labels Dec 28, 2017

jtojnar reviewed Dec 28, 2017

View reviewed changes

jtojnar requested changes Dec 28, 2017

View reviewed changes

GrahamcOfBorg reviewed Dec 28, 2017

View reviewed changes

jtojnar changed the title ~~chainer: init at 3.2.0~~ pythonPackages.chainer: init at 3.2.0 Dec 28, 2017

FRidh requested changes Dec 28, 2017

View reviewed changes

jtojnar approved these changes Dec 28, 2017

View reviewed changes

jtojnar force-pushed the add/chainer-add branch from bf7b139 to 06c0ebc Compare December 28, 2017 16:45

hyphon81 added 6 commits December 28, 2017 17:46

maintainers: Add hyphon81

50a6403

nccl: init at 1.3.4-1

05573d3

pythonPackages.fastrlock: init at 0.3

a9379a6

pythonPackages.filelock: init at 2.0.13

e592101

pythonPackages.cupy: init at 2.2.0

0184f60

pythonPackages.chainer: init at 3.2.0

7ccceea

jtojnar force-pushed the add/chainer-add branch from 06c0ebc to 7ccceea Compare December 28, 2017 16:47

jtojnar approved these changes Dec 28, 2017

View reviewed changes

jtojnar merged commit 7ccceea into NixOS:master Dec 28, 2017

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.chainer: init at 3.2.0 #33123

pythonPackages.chainer: init at 3.2.0 #33123

hyphon81 commented Dec 28, 2017

jtojnar Dec 28, 2017

jtojnar Dec 28, 2017

jtojnar Dec 28, 2017

jtojnar Dec 28, 2017

jtojnar Dec 28, 2017

hyphon81 Dec 28, 2017

jtojnar Dec 28, 2017

hyphon81 Dec 28, 2017

jtojnar Dec 28, 2017

hyphon81 Dec 28, 2017

jtojnar Dec 28, 2017

jtojnar Dec 28, 2017 •

edited

jtojnar Dec 28, 2017

hyphon81 Dec 28, 2017

jtojnar Dec 28, 2017

jtojnar commented Dec 28, 2017

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

jtojnar commented Dec 28, 2017

FRidh Dec 28, 2017

jtojnar commented Dec 28, 2017

pythonPackages.chainer: init at 3.2.0 #33123

pythonPackages.chainer: init at 3.2.0 #33123

Conversation

hyphon81 commented Dec 28, 2017

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtojnar Dec 28, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtojnar commented Dec 28, 2017

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

jtojnar commented Dec 28, 2017

Choose a reason for hiding this comment

jtojnar commented Dec 28, 2017

jtojnar Dec 28, 2017 •

edited