python.pkgs.tensorflow: 1.14.0 -> 1.15.0 #71282

timokau · 2019-10-17T11:52:14Z

Motivation for this change

Continuation of #70910.
fixes #71578.

Things done

timokau · 2019-10-17T11:53:39Z

I have decided to keep the default to no-cuda for now, unless there are any objections.

timokau · 2019-10-17T11:57:06Z

This should also fix the tensorflow build, which was broken by #69252.

timokau · 2019-10-17T20:00:43Z

Unfortunately the build still doesn't quite work with bazel 1.0. Upstream is tracking this here: tensorflow/tensorflow#33240

flokli · 2019-10-21T21:10:45Z

Upstream is working on a fix in tensorflow/tensorflow#33415.

coreyoconnor · 2019-10-23T21:08:59Z

Did a quick pass at applying this patch. Unfortunately, I'm unable to build this with upstream's fix applied. Will continue looking into it some later day.

Adding to here FYI:

coreyoconnor@8f226c8

on that nixpkgs fork the build fails with:

/nix/store/jy6zl5dkw5rcjczm1arjqxz95l9d40i7-binutils-2.31.1/bin/ld: bazel-out/host/bin/tensorflow/core/libop_gen_lib.a(op_gen_lib.o): in function `std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tensorflow::ApiDef>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tensorflow::ApiDef> >, std::__deta
il::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail
::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<
char>, std::allocator<char> > const, tensorflow::ApiDef>, true>*)':
op_gen_lib.cc:(.text._ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_N10tensorflow6ApiDefEESaISA_ENSt8__detail10_Select1stESt8equal_toIS5_ESt4hashIS5_ENSC_18_Mod_range_hashingENSC_20_Default_ranged_hashENSC_20_Prime_reha
sh_policyENSC_17_Hashtable_traitsILb1ELb0ELb1EEEE21_M_insert_unique_nodeEmmPNSC_10_Hash_nodeISA_Lb1EEE[_ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_N10tensorflow6ApiDefEESaISA_ENSt8__detail10_Select1stESt8equal_toIS5_
ESt4hashIS5_ENSC_18_Mod_range_hashingENSC_20_Default_ranged_hashENSC_20_Prime_rehash_policyENSC_17_Hashtable_traitsILb1ELb0ELb1EEEE21_M_insert_unique_nodeEmmPNSC_10_Hash_nodeISA_Lb1EEE]+0xe1): undefined reference to `tensorflow::ApiDef::~ApiDef()'
/nix/store/jy6zl5dkw5rcjczm1arjqxz95l9d40i7-binutils-2.31.1/bin/ld: bazel-out/host/bin/tensorflow/core/libop_gen_lib.a(op_gen_lib.o): in function `std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, s
td::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tensorflow::ApiDef>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tensorflow::ApiDef> >, st
d::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std:
:__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
op_gen_lib.cc:(.text._ZNSt8__detail9_Map_baseINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS6_N10tensorflow6ApiDefEESaISB_ENS_10_Select1stESt8equal_toIS6_ESt4hashIS6_ENS_18_Mod_range_hashingENS_20_Default_ranged_hashENS_20_Prime_rehash_po
licyENS_17_Hashtable_traitsILb1ELb0ELb1EEELb1EEixERS8_[_ZNSt8__detail9_Map_baseINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS6_N10tensorflow6ApiDefEESaISB_ENS_10_Select1stESt8equal_toIS6_ESt4hashIS6_ENS_18_Mod_range_hashingENS_20_Default
_ranged_hashENS_20_Prime_rehash_policyENS_17_Hashtable_traitsILb1ELb0ELb1EEELb1EEixERS8_]+0xa8): undefined reference to `tensorflow::ApiDef::ApiDef()'
/nix/store/jy6zl5dkw5rcjczm1arjqxz95l9d40i7-binutils-2.31.1/bin/ld: bazel-out/host/bin/tensorflow/core/libop_gen_lib.a(op_gen_lib.o): in function `std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tensorflow::ApiDef>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tensorflow::ApiDef> >, std::__deta
il::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail
::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable()':
op_gen_lib.cc:(.text._ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_N10tensorflow6ApiDefEESaISA_ENSt8__detail10_Select1stESt8equal_toIS5_ESt4hashIS5_ENSC_18_Mod_range_hashingENSC_20_Default_ranged_hashENSC_20_Prime_reha
sh_policyENSC_17_Hashtable_traitsILb1ELb0ELb1EEEED2Ev[_ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_N10tensorflow6ApiDefEESaISA_ENSt8__detail10_Select1stESt8equal_toIS5_ESt4hashIS5_ENSC_18_Mod_range_hashingENSC_20_Defa
ult_ranged_hashENSC_20_Prime_rehash_policyENSC_17_Hashtable_traitsILb1ELb0ELb1EEEED5Ev]+0x28): undefined reference to `tensorflow::ApiDef::~ApiDef()'
/nix/store/jy6zl5dkw5rcjczm1arjqxz95l9d40i7-binutils-2.31.1/bin/ld: bazel-out/host/bin/tensorflow/core/libop_gen_lib.a(op_gen_lib.o): in function `tensorflow::ApiDefMap::ApiDefMap(tensorflow::OpList const&)':
op_gen_lib.cc:(.text._ZN10tensorflow9ApiDefMapC2ERKNS_6OpListE+0xcc): undefined reference to `tensorflow::ApiDef::ApiDef()'
....

coreyoconnor · 2019-10-24T05:10:04Z

Ignore my prior comment. Used an out-of-date version of this pull request. Re-testing.

coreyoconnor · 2019-10-24T16:39:49Z

Re-tested using master + this patch, with adjustments to the deps SHA. On our build farm, both cuda and non-cuda fail to compile with similar link errors:

bazel-out/host/bin/tensorflow/python/gen_nn_ops_py_wrappers_cc: symbol lookup error: /build/output/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/../../_solib_k8/_U_S_Stensorflow_Spython_Cgen_Unn_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so.1: undefined symbol: _ZN10tensorflow16ProtoDebugStringB5cxx11ERKNS_16DeviceAttributesE

Unfortunate that the tensorflow build is a monolith. :( Nothing for nixpkgs to resolve afaict. Kinda surprising tho. Pain to test/build.

timor · 2019-11-11T15:24:21Z

@coreyoconnor This seems to stem from protobuf being an alias for protobuf3_7. ~~Tensorflow-1.15.0 does seem to require protobuf3_6 though.~~

Relevant upstream commit: tensorflow/tensorflow@f92a0e6

Edit... or maybe not. I have a hard time telling which code is from which project here... Actually, it seems to be that they actually upgraded protobuf to 3.8.0 in v1.15.0, but that commit above is not present in 1.15.0

timokau · 2019-11-12T09:41:49Z

Thanks to @scentini in tensorflow/tensorflow#33415, it builds now!

Things left to do:

some sanity checking
fill in the cuda hash

If anybody else wants to do some testing, now is the time.

… 1.14.0 -> 1.15.0

The tensorflow build was broken by the bazel 1.0 update in 73eb01b.

coreyoconnor · 2019-11-12T23:14:20Z

bazel-out/host/bin/tensorflow/python/gen_nn_ops_py_wrappers_cc: symbol lookup error: /build/output/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/../../_solib_k8/_U_S_Stensorflow_Spython_Cgen_Unn_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so.1: undefined symbol: _ZN10tensorflow16ProtoDebugStringB5cxx11ERKNS_16DeviceAttributesE

this issue is resolved with timokau changes. Unfortunately, I missed the latest patch and ran into the gast error. XD

Testing again.

timokau · 2019-11-12T23:20:24Z

Yeah, sorry about that. I made some mistake during fixup before push. Should work now.

coreyoconnor · 2019-11-12T23:32:50Z

@timokau no problem!

Subsequent build failed on this

      # for some reason some of the required libs are in the targets/x86_64-linux                                                                                                             
      # directory; not sure why but this works around it                                                                                                                                      
      "${cudatoolkit}/targets/${stdenv.system}"

I'm building using cudatoolkit 10. Which is not the default in all-packages. I wonder if this workaround is specific to cudatoolkit 9?

coreyoconnor · 2019-11-13T01:41:42Z

Success! Compiled without issue with cudatoolkit_10 (once I removed that workaround). So I presume yes: that is specific to cudatoolkit_9.

timokau · 2019-11-13T08:41:11Z

Thanks for testing! I don't quite understand what you're saying though. So the workaround is not necessary for a newer cudatoolkit? That's good, but why does it fail? If the workaround is not necessary, it still shouldn't do any harm right?

Now needed at runtime with the python2 build.

timokau · 2019-11-13T16:11:12Z

Merging now, since this has been broken on master for so long. Any fixups can be made in followup PRs. For example, python2 is still broken. It's probably not too difficult to fix, but I don't plan to invest more time in this for now.

coreyoconnor · 2019-11-13T17:23:58Z

Correct: the work around is not needed for newer cuda. The workaround actually instructs nix to copy that specific path to the store. Which does not exist on newer cuda toolkit. Thus it will fail due to not finding the path. (On mobile)

…

On Wed, Nov 13, 2019, 12:41 AM Timo Kaufmann ***@***.***> wrote: Thanks for testing! I don't quite understand what you're saying though. So the workaround is not necessary for a newer cudatoolkit? That's good, but why does it fail? If the workaround is not necessary, it still shouldn't do any harm right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71282?email_source=notifications&email_token=AAAIMDPASHOEV2BJNL4YQJ3QTO4TDA5CNFSM4JBYKW7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED5KQTQ#issuecomment-553297998>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIMDICQVDEYWLWQ2FTU3LQTO4TDANCNFSM4JBYKW7A> .

coreyoconnor · 2019-11-13T17:25:11Z

Oh: the default cudtoolkit is 9. Which this does compile fine for. Fixing for cuda 10 can wait :) definitely not a merge blocker

…

On Wed, Nov 13, 2019, 8:11 AM Timo Kaufmann ***@***.***> wrote: Merging now, since this has been broken on master for so long. Any fixups can be made in followup PRs. For example, python2 is still broken. It's probably not too difficult to fix, but I don't plan to invest more time in this for now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71282?email_source=notifications&email_token=AAAIMDN2JQ4VBWBDTZL7JJLQTQRKTA5CNFSM4JBYKW7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED6VF6Y#issuecomment-553472763>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIMDOM6T7JHJPYZFZ5BBDQTQRKTANCNFSM4JBYKW7A> .

timokau · 2019-11-13T17:50:06Z

Oh, so the path doesn't exist. Its weird that the workaround is necessary at all. The best solution would probably be to fix the cudatoolkit 9 build to make the workaround unnecessary for both versions.

Do feel free to create a PR to fix this either in the tensorflow build or in the cudatoolkit build. If you do, ping me and I'll review (to the best of my ability, since I don't have a nvida card) :)

coreyoconnor · 2019-11-13T18:13:02Z

@timokau

will do! Thanks again for figuring this out :)

timokau requested a review from FRidh as a code owner October 17, 2019 11:52

timokau changed the title ~~Tensorflow 1.15.0~~ python.pkgs.tensorflow: 1.14.0 -> 1.15.0 Oct 17, 2019

timokau mentioned this pull request Oct 17, 2019

python.pkgs.tensorflow: 1.15.0 -> 2.0.0 #70910

Closed

ofborg bot added the 6.topic: python label Oct 17, 2019

ofborg bot requested a review from abbradar October 17, 2019 12:04

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Oct 17, 2019

timokau added the 2.status: work-in-progress label Oct 17, 2019

timokau changed the title ~~python.pkgs.tensorflow: 1.14.0 -> 1.15.0~~ [WIP] python.pkgs.tensorflow: 1.14.0 -> 1.15.0 Oct 17, 2019

timokau added the 2.status: wait-for-upstream label Oct 17, 2019

timokau force-pushed the tensorflow-1.15.0 branch from 913b48f to 2fdfb1a Compare October 17, 2019 20:12

flokli mentioned this pull request Oct 21, 2019

python3Packages.tensorflow build is broken on master #71578

Closed

tbenst mentioned this pull request Oct 30, 2019

pythonPackages.deeplabcut: init at 2.1.6 #61253

Closed

10 tasks

timokau force-pushed the tensorflow-1.15.0 branch from 2fdfb1a to 2583a43 Compare November 12, 2019 09:41

timokau requested a review from jonringer as a code owner November 12, 2019 09:41

timokau force-pushed the tensorflow-1.15.0 branch 2 times, most recently from 7352080 to 62108eb Compare November 12, 2019 09:55

timokau added 3 commits November 12, 2019 23:42

python.pkgs.{tensorflow,tensorflow-estimator,tensorflow-tensorboard}:…

152d2fd

… 1.14.0 -> 1.15.0

python.pkgs.tensorflow: fix build with bazel 1.0

1fbd307

The tensorflow build was broken by the bazel 1.0 update in 73eb01b.

python.pkgs.tensorflow: fix build with newer gast

4fb7831

timokau force-pushed the tensorflow-1.15.0 branch from 62108eb to 4fb7831 Compare November 12, 2019 22:43

python.pkgs.tensorflow: add functools32 dependency

6832f1d

Now needed at runtime with the python2 build.

timokau force-pushed the tensorflow-1.15.0 branch from 0c31f7c to 5162862 Compare November 13, 2019 15:17

timokau added 2 commits November 13, 2019 16:18

python.pkgs.tensorflow: add more sanity checks

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

e4378d4

python.pkgs.tensorflow: mark as broken for python2

8e382a7

timokau force-pushed the tensorflow-1.15.0 branch from 5162862 to 8e382a7 Compare November 13, 2019 15:19

timokau changed the title ~~[WIP] python.pkgs.tensorflow: 1.14.0 -> 1.15.0~~ python.pkgs.tensorflow: 1.14.0 -> 1.15.0 Nov 13, 2019

timokau removed 2.status: wait-for-upstream 2.status: work-in-progress labels Nov 13, 2019

timokau merged commit 0ae46ca into NixOS:master Nov 13, 2019

timokau deleted the tensorflow-1.15.0 branch November 13, 2019 16:11

veprbl mentioned this pull request Nov 26, 2019

tensorflow-bin build fails #73703

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python.pkgs.tensorflow: 1.14.0 -> 1.15.0 #71282

python.pkgs.tensorflow: 1.14.0 -> 1.15.0 #71282

timokau commented Oct 17, 2019 •

edited by flokli

Loading

timokau commented Oct 17, 2019

timokau commented Oct 17, 2019

timokau commented Oct 17, 2019

flokli commented Oct 21, 2019

coreyoconnor commented Oct 23, 2019

coreyoconnor commented Oct 24, 2019

coreyoconnor commented Oct 24, 2019

timor commented Nov 11, 2019 •

edited

Loading

timokau commented Nov 12, 2019 •

edited

Loading

coreyoconnor commented Nov 12, 2019

timokau commented Nov 12, 2019

coreyoconnor commented Nov 12, 2019

coreyoconnor commented Nov 13, 2019

timokau commented Nov 13, 2019

timokau commented Nov 13, 2019

coreyoconnor commented Nov 13, 2019 via email

coreyoconnor commented Nov 13, 2019 via email

timokau commented Nov 13, 2019

coreyoconnor commented Nov 13, 2019

python.pkgs.tensorflow: 1.14.0 -> 1.15.0 #71282

python.pkgs.tensorflow: 1.14.0 -> 1.15.0 #71282

Conversation

timokau commented Oct 17, 2019 • edited by flokli Loading

Motivation for this change

Things done

timokau commented Oct 17, 2019

timokau commented Oct 17, 2019

timokau commented Oct 17, 2019

flokli commented Oct 21, 2019

coreyoconnor commented Oct 23, 2019

coreyoconnor commented Oct 24, 2019

coreyoconnor commented Oct 24, 2019

timor commented Nov 11, 2019 • edited Loading

timokau commented Nov 12, 2019 • edited Loading

coreyoconnor commented Nov 12, 2019

timokau commented Nov 12, 2019

coreyoconnor commented Nov 12, 2019

coreyoconnor commented Nov 13, 2019

timokau commented Nov 13, 2019

timokau commented Nov 13, 2019

coreyoconnor commented Nov 13, 2019 via email

coreyoconnor commented Nov 13, 2019 via email

timokau commented Nov 13, 2019

coreyoconnor commented Nov 13, 2019

timokau commented Oct 17, 2019 •

edited by flokli

Loading

timor commented Nov 11, 2019 •

edited

Loading

timokau commented Nov 12, 2019 •

edited

Loading