Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensorflow1.3 dependencies #28530

Closed
wants to merge 4 commits into from
Closed

Conversation

jyp
Copy link
Contributor

@jyp jyp commented Aug 24, 2017

Motivation for this change

This branch prepares for tensorflow 1.3 by adding the necessary dependencies.
Unfortunately, this branch does not add tensorflow 1.3 itself because of a circular dependency between tensorflow and tensorflow_tensorboard.

Reviewers: please advise as how to deal with such a circular dependency between python packages. The manual https://nixos.org/nixpkgs/manual/#how-to-solve-circular-dependencies prescribe to override the packages to remove the offending dependency however it is silent on how this should be done.

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@jyp jyp requested a review from FRidh as a code owner August 24, 2017 09:04
@jyp jyp force-pushed the tensorflow1.3-dependencies branch from d4aadb6 to d831b65 Compare August 24, 2017 09:30
@@ -2089,6 +2089,8 @@ in {

httpserver = callPackage ../development/python-modules/httpserver {};

bleach1_5 = callPackage ../development/python-modules/bleach/1.5.nix {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we keep only one version of a package, not more. What do you need it for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a "tensorflow-tensorboard" dependency. Perhaps we won't need it if we decide to drop tensorboard, see my comment on the PR.

}:

buildPythonPackage rec {
name = "${pname}-${version}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 spaces instead of four

}:

buildPythonPackage rec {
pname = "bleach";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 spaces instead of four

description = "NVIDIA CUDA Deep Neural Network library (cuDNN)";
homepage = https://developer.nvidia.com/cudnn;
license = stdenv.lib.licenses.unfree;
maintainers = with maintainers; [ mdaiter ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you mean the license, nvidia states: "If the SOFTWARE is provided in source form, Licensee may not modify or create derivative works of the SOFTWARE." So I'd say definitely unfree.

If you mean the maintainer then I have no opinion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The maintainer. You add a new expression (I suppose you copied it) adding someone else as maintainer. You've asked the person whether they're willing to maintain this expression?

@FRidh
Copy link
Member

FRidh commented Aug 24, 2017

Unfortunately, this branch does not add tensorflow 1.3 itself because of a circular dependency between tensorflow and tensorflow_tensorboard.

Reviewers: please advise as how to deal with such a circular dependency between python packages. The manual https://nixos.org/nixpkgs/manual/#how-to-solve-circular-dependencies prescribe to override the packages to remove the offending dependency however it is silent on how this should be done.

Are they build from source or are you installing binary wheels?

In case of the former, it typically means overriding the install_requires that is passed to setup() in setup.py. In case of the latter there is no solution, as far as I know.

@jyp
Copy link
Contributor Author

jyp commented Aug 24, 2017

Regarding the circular dependency: tensorflow and tensorboard are provided as wheels. I found a workaround though: one can supply --no-deps to pip. Then it won't check the dependencies at install time.

But then again, I cannot install both tensorflow and tensorboard because of the following collision:

 /nix/store/wbw4k3ih95qix8fqrwv35c32s54laj51-python3.5-tensorflow-1.3.0/lib/python3.5/site-packages/external/__pycache__/__init__.cpython-35.pyc' and
/nix/store/glc905np5xqpj5pfy0mph12ihxmvrwiv-python3.5-tensorflow_tensorboard-0.1.4/lib/python3.5/site-packages/external/__pycache__/__init__.cpython-35.pyc

Tensorflow is still usable without tensorboard, so I would propose to not install tensorboard for the time being.

@jyp jyp force-pushed the tensorflow1.3-dependencies branch from bfdcdc2 to c7555dd Compare August 24, 2017 12:19
@jyp
Copy link
Contributor Author

jyp commented Aug 24, 2017

@FRidh
I have apply the style changes. bleach 1.5 may be dropped in case we go with my proposal (above comment).

@jyp jyp force-pushed the tensorflow1.3-dependencies branch from c7555dd to a4f836c Compare August 24, 2017 12:25
@FRidh
Copy link
Member

FRidh commented Aug 24, 2017

@jyp if it checks for dependencies then its likely that it is also needed during runtime. Have you checked upstream what they say about requiring bleach?

@jyp
Copy link
Contributor Author

jyp commented Aug 24, 2017

@FRidh I don't think I made myself so clear :)

I'm thinking to remove tensorboard from the dependencies of tensorflow (core). Tensorboard is the visualization engine of tensorflow. According to my tests it isn't required to run tensorflow models --- only to visualize the results. So as far as I understand tensorboard is not a necessary component and should be dropped from the dependencies of tensorflow --- even upstream.

Now, we have then the opportunity to remove tensorboard entirely from nixpkgs; and in turn its dependencies, including bleach 1.5.

@jyp jyp force-pushed the tensorflow1.3-dependencies branch from a4f836c to a8afe3a Compare August 28, 2017 19:01
@jyp jyp force-pushed the tensorflow1.3-dependencies branch from a8afe3a to e8a20fb Compare August 28, 2017 19:22
@jyp
Copy link
Contributor Author

jyp commented Aug 30, 2017

@FRidh Do you think that this PR could be accepted in the current state? I've dropped the extra bleach version. What remains is I believe rather uncontroversial:

  • update version of markdown
  • add newest protobuf version
  • add newest cudnn version
  • add weakref backport

@FRidh
Copy link
Member

FRidh commented Sep 6, 2017

I've added these commits to #29009.

@FRidh FRidh closed this Sep 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants