python37Packages.tensorflow-bin: 1.14.0 -> 2.1.0 #81278

cdepillabout · 2020-02-28T09:30:15Z

Motivation for this change

This bumps python37Packages.tensorflow-bin to version 2.1.0.

It also bumps some of the dependencies of tensorflow-bin to the required versions:

tensorflow-tensorboard: 1.15.0 -> 2.1.0
tensorflow-estimator: 1.15.1 -> 2.1.0

With this PR, tensorflow-bin builds correctly, and appears to be able to be used correctly with python37Packages.

I've tested it with the following simple python script (from https://www.tensorflow.org/tutorials/quickstart/beginner):

from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf

mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer='adam', loss=loss_fn, metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test,  y_test, verbose=2)

I've only tested this on Linux without CUDA. I would appreciate if someone could try this out on Mac or with CUDA enabled.

I'm a beginner with both Tensorflow and Python, so please let me know if I am doing anything weird here.

Things done

pkgs/development/python-modules/tensorflow/prefetcher.sh

pkgs/top-level/python-packages.nix

cdepillabout · 2020-02-28T09:46:05Z

Here are two related PRs:

tensorflow-tensorboard: 1.14.0 -> 2.0.0 #71890: bumps tensorflow-tensorboard to 2.0.0, but it should be superceded by this current PR.
python.pkgs.tensorflow: 1.15.0 -> 2.0.0 #70910: bumps tensorflow to 2.0.0. This isn't directly related to this current PR, but people who are following that will probably be interested in this current PR as well. cc @tbenst @danieldk @mjlbach @Zhen-hao @Ericson2314

cdepillabout · 2020-02-28T09:53:11Z

I'm currently in the process of testing the following:

Linux, non-cuda:
- python36Packages.tensorflow-bin (I've confirmed this works)
- python35Packages.tensorflow-bin (this fails because the tests fail for opt_einsum-3.1.0, which is a direct dependency of tensorflow-bin. As far as I can tell, opt_einsum-3.1.0 doesn't support python35. I created an issue upstream: modify python_requires in setup.py to require python 3.6+ dgasmith/opt_einsum#126)
- python27Packages.tensorflow-bin (this fails while running the tests for google-auth-oauthlib-0.4.1, which is a dependency of tensorflow-tensorboard, which is used by tensorflow-bin)

cdepillabout · 2020-02-28T09:55:16Z

@GrahamcOfBorg build python37Packages.tensorflow-bin python36Packages.tensorflow-bin python35Packages.tensorflow-bin python27Packages.tensorflow-bin

FRidh

Did not test this but the change looks good to me.

…nsorflow-2.1.0

cdepillabout · 2020-02-28T12:31:27Z

I forgot to update the hash for tensorflow-tensorboard for python27, so I just updated it and force-pushed.

veprbl · 2020-02-28T12:39:07Z

Shouldn't update to pythonPackages.tensorflow-tensorboard and pythonPackages.tensorflow-estimator break the pythonPackages.tensorflow unless we don't bump it as well?

cdepillabout · 2020-02-28T12:44:51Z

@veprbl I was under the impression that pythonPackages.tensorflow was already broken.

When I try to build it on master, I get the following message:

$ nix-build -A python37Packages.tensorflow
error: Package ‘openssl-1.0.2u’ in /home/illabout/git/nixpkgs/pkgs/development/libraries/openssl/default.nix:130 is marked as insecure, refusing to evaluate.

Known issues:
 - Support for OpenSSL 1.0.2 ended with 2019.

You can install it anyway by whitelisting this package, using the
following methods:
...

I don't think this should hold back updating tensorflow-bin.

Although, it would also be great to get an up-to-date version of pythonPackages.tensorflow. It is unfortunately out-of-scope for this PR though.

jonringer · 2020-02-28T16:34:34Z

@veprbl I was under the impression that pythonPackages.tensorflow was already broken.

This is my understanding as well, I'm fine with surpassing a broken build.

tbenst · 2020-03-01T22:07:01Z

tensorflow is fixed pending merge of two PRs: #80379 (comment).

tensorflow 2 is sadly not fully backward-compatible with tensorflow 1, and there are a number of libraries / applications that are not moving. I’d love to see both versions in nixpkgs.

andir · 2020-03-02T16:19:03Z

This now broke the non-bin package of tensorflow. Some of us were using it but others (like me) were trying to unbreak the edge-cases I am hitting.

TF 1.15 might have been broken in a few ways. With this PR merged we now need to unbreak even more.. I guess I'll just go about adding a second version of TF estimator for the older TF 1.15.

FRidh · 2020-03-02T16:26:39Z

Those that need tensorflow and related packages need to cooperate a bit more, because with all these different PR's touching it, it is hard to follow.

timokau · 2020-03-02T22:31:24Z

Since there seem to be many people who are very interested in keeping it working, it would be nice if some of them could add themselves as additional maintainers. Due to time constraints maintaining it isn't a very high priority for me personally right now.

cdepillabout · 2020-03-03T04:28:10Z

@andir Sorry for breaking the other tensorflow with this.

Could we merge in #70910 (after possibly updating it to 2.1.0)? That way we wouldn't have to carry around multiple versions of tensorflow-estimator (which I thought was strictly not allowed in the python package set).

Zhen-hao · 2020-04-04T10:07:34Z

hi, what is the new tensorflowWithCuda for 2.1?
Is it built in tensorflow_2?

Ericson2314 · 2020-04-04T18:07:44Z

Look at the old ones definition? All I know of is a cudaSupport flag on 1.x and 2.x.

Zhen-hao · 2020-04-04T19:30:03Z

you are right.

  tensorflowWithCuda = self.tensorflow.override {
    cudaSupport = true;
  };

cdepillabout requested review from abbradar, timokau and jonringer February 28, 2020 09:30

cdepillabout requested a review from FRidh as a code owner February 28, 2020 09:30

cdepillabout commented Feb 28, 2020

View reviewed changes

pkgs/development/python-modules/tensorflow/prefetcher.sh Show resolved Hide resolved

ofborg bot added the 6.topic: python label Feb 28, 2020

cdepillabout commented Feb 28, 2020

View reviewed changes

pkgs/top-level/python-packages.nix Show resolved Hide resolved

cdepillabout force-pushed the tensorflow-bin-2.1.0 branch from c04cdd4 to ae44863 Compare February 28, 2020 09:52

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Feb 28, 2020

FRidh approved these changes Feb 28, 2020

View reviewed changes

cdepillabout added 4 commits February 28, 2020 21:27

python3Packages.tensorflow-tensorboard: 1.15.0 -> 2.1.0

43444a4

python3Packages.tensorflow-estimator: 1.15.1 -> 2.1.0

4957b56

python3Packages.protobuf: bump protobuf dep to from 3.7 to 3.8 for te…

948155c

…nsorflow-2.1.0

python3Packages.tensorflow-bin: 1.14.0 -> 2.1.0

cf5d1a7

cdepillabout force-pushed the tensorflow-bin-2.1.0 branch from 7b0c1c8 to cf5d1a7 Compare February 28, 2020 12:28

FRidh merged commit 9e8dea7 into NixOS:master Mar 2, 2020

cdepillabout deleted the tensorflow-bin-2.1.0 branch March 2, 2020 08:38

andir mentioned this pull request Mar 2, 2020

buildBazelPackage: poison all the fixed output derivations #80953

Merged

8 tasks

cdepillabout mentioned this pull request Mar 3, 2020

tensorflow-tensorboard: 1.14.0 -> 2.0.0 #71890

Closed

10 tasks

marsam mentioned this pull request Jul 2, 2020

use latest protobuf (3.12), remove overrides where possible #91446

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python37Packages.tensorflow-bin: 1.14.0 -> 2.1.0 #81278

python37Packages.tensorflow-bin: 1.14.0 -> 2.1.0 #81278

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Feb 28, 2020

FRidh left a comment

cdepillabout commented Feb 28, 2020

veprbl commented Feb 28, 2020

cdepillabout commented Feb 28, 2020 •

edited

jonringer commented Feb 28, 2020

tbenst commented Mar 1, 2020

andir commented Mar 2, 2020

FRidh commented Mar 2, 2020

timokau commented Mar 2, 2020

cdepillabout commented Mar 3, 2020 •

edited

Zhen-hao commented Apr 4, 2020 •

edited

Ericson2314 commented Apr 4, 2020

Zhen-hao commented Apr 4, 2020

python37Packages.tensorflow-bin: 1.14.0 -> 2.1.0 #81278

python37Packages.tensorflow-bin: 1.14.0 -> 2.1.0 #81278

Conversation

cdepillabout commented Feb 28, 2020 • edited

Motivation for this change

Things done

cdepillabout commented Feb 28, 2020 • edited

cdepillabout commented Feb 28, 2020 • edited

cdepillabout commented Feb 28, 2020

FRidh left a comment

Choose a reason for hiding this comment

cdepillabout commented Feb 28, 2020

veprbl commented Feb 28, 2020

cdepillabout commented Feb 28, 2020 • edited

jonringer commented Feb 28, 2020

tbenst commented Mar 1, 2020

andir commented Mar 2, 2020

FRidh commented Mar 2, 2020

timokau commented Mar 2, 2020

cdepillabout commented Mar 3, 2020 • edited

Zhen-hao commented Apr 4, 2020 • edited

Ericson2314 commented Apr 4, 2020

Zhen-hao commented Apr 4, 2020

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Feb 28, 2020 •

edited

cdepillabout commented Mar 3, 2020 •

edited

Zhen-hao commented Apr 4, 2020 •

edited