Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpc, pythonPackges.grpcio, pythonPackages.grpcio-tools: update to 1.16.1 #50672

Merged
merged 4 commits into from Nov 27, 2018

Conversation

lopsided98
Copy link
Contributor

Motivation for this change

Updates grpc and related packages to 1.16.1:

  • grpc: 1.15.0 -> 1.16.1
  • pythonPackages.grpcio: 1.15.0 ->1.16.1
  • pythonPackages.grpcio-tools: 1.14.2 -> 1.16.1

I removed the grpc dependency from grpcio-tools, because it is not used.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@worldofpeace
Copy link
Contributor

@GrahamcOfBorg build grpc pythonPackages.grpcio python3Packages.grpcio pythonPackages.grpcio-tools python3Packages.grpcio-tools python3Packages.tensorflow pythonPackages.tensorflow

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: grpc, pythonPackages.grpcio, python3Packages.grpcio, pythonPackages.grpcio-tools, python3Packages.grpcio-tools, pythonPackages.tensorflow

The following builds were skipped because they don't evaluate on aarch64-linux: python3Packages.tensorflow

Partial log (click to expand)

Successfully installed grpcio-tools-1.16.1
/build/grpcio-tools-1.16.1
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ry731msk6pg91z8w4f00466x91nnz3fg-python2.7-grpcio-tools-1.16.1
shrinking /nix/store/ry731msk6pg91z8w4f00466x91nnz3fg-python2.7-grpcio-tools-1.16.1/lib/python2.7/site-packages/grpc_tools/_protoc_compiler.so
strip is /nix/store/6dpnd5aniypn8124mmy8f88s4mq2zl07-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/ry731msk6pg91z8w4f00466x91nnz3fg-python2.7-grpcio-tools-1.16.1/lib
patching script interpreter paths in /nix/store/ry731msk6pg91z8w4f00466x91nnz3fg-python2.7-grpcio-tools-1.16.1
checking for references to /build in /nix/store/ry731msk6pg91z8w4f00466x91nnz3fg-python2.7-grpcio-tools-1.16.1...
error: build of '/nix/store/3gqx3df2phy3k6q4x36026d5n6jj1vk1-python2.7-tensorflow-1.11.0.drv', '/nix/store/9s1waiq82jhxickkalgkj7s9hj0kg4ys-python3.7-grpcio-1.16.1.drv', '/nix/store/nk79bwv061kgvkjd6jiynv75hlk344bm-python3.7-grpcio-tools-1.16.1.drv' failed

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: grpc, pythonPackages.grpcio, python3Packages.grpcio, pythonPackages.grpcio-tools, python3Packages.grpcio-tools, pythonPackages.tensorflow

The following builds were skipped because they don't evaluate on x86_64-linux: python3Packages.tensorflow

Partial log (click to expand)

Successfully installed grpcio-tools-1.16.1
/build/grpcio-tools-1.16.1
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/caf1n3l6w5l12cv9dlab4gwfpxh02xbb-python2.7-grpcio-tools-1.16.1
shrinking /nix/store/caf1n3l6w5l12cv9dlab4gwfpxh02xbb-python2.7-grpcio-tools-1.16.1/lib/python2.7/site-packages/grpc_tools/_protoc_compiler.so
strip is /nix/store/rpbg8gmqxhz8g61p1plz5d2srs84pvmv-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/caf1n3l6w5l12cv9dlab4gwfpxh02xbb-python2.7-grpcio-tools-1.16.1/lib
patching script interpreter paths in /nix/store/caf1n3l6w5l12cv9dlab4gwfpxh02xbb-python2.7-grpcio-tools-1.16.1
checking for references to /build in /nix/store/caf1n3l6w5l12cv9dlab4gwfpxh02xbb-python2.7-grpcio-tools-1.16.1...
error: build of '/nix/store/06iyc3l81dkhvzv02c3z3zz38l9bdajp-python3.7-grpcio-1.16.1.drv', '/nix/store/f18xvzxgidbnpc8v6xqy3kl9263r5xya-python3.7-grpcio-tools-1.16.1.drv' failed

@lopsided98
Copy link
Contributor Author

I added a commit which applies protocolbuffers/protobuf#4862 to make protobuf compatible with Python 3.7.

@worldofpeace
Copy link
Contributor

@GrahamcOfBorg build grpc pythonPackages.grpcio python3Packages.grpcio pythonPackages.grpcio-tools python3Packages.grpcio-tools pythonPackages.protobuf python3Packages.protobuf

@worldofpeace
Copy link
Contributor

@GrahamcOfBorg build pythonPackages.tensorflow

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: pythonPackages.tensorflow

Partial log (click to expand)

cannot link '/nix/store/.links/1rzsr21khppp3kc33qa3x45b645nmnwazv9w4a60s1sj6xiszhy1' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/util/mkl_util.h': No space left on device
cannot link '/nix/store/.links/0020qq6kxd2vicqckx94kmli1awryysbkzfc0qp7yy2124gh410z' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/python/ops/template.pyc': No space left on device
cannot link '/nix/store/.links/03jkvkmkd739rjk5zri4ckyxfnzwyc6kb37bbvdhgxxvwmnp82j5' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/core/protobuf/debug_pb2.pyc': No space left on device
cannot link '/nix/store/.links/14j4ls6f5qddl5z891n6a1kfpf25wknsfzv2fzqjcqscw5003jvh' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/readers.pyc': No space left on device
cannot link '/nix/store/.links/1vawl2bj65zg55wg97shr3lv5jazpcag0ykjvlilsjclkk8bkjkj' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/contrib/gan/python/estimator/python/gan_estimator_impl.py': No space left on device
cannot link '/nix/store/.links/19665cqczrmvl3hjlvsyj40r5mcwa6c69jwy0kahw5knr22s8z3y' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/contrib/gan/__init__.pyc': No space left on device
cannot link '/nix/store/.links/1glvih5anasrxjldzbkszg0kkgwd07f9kd4gc9kc46qjm9yrnnws' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/data/dataset.pyc': No space left on device
cannot link '/nix/store/.links/0v6vrbz4vr8xapa5i8hg51c7y1zxz296zdxal6mxml9pxzhrw19b' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/contrib/boosted_trees/estimator_batch/distillation_loss.pyc': No space left on device
cannot link '/nix/store/.links/030f2qjy27r4mg6w8l55anzx997wln33p6hqigvc2lg20a6p0kpl' to '/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0/lib/python2.7/site-packages/tensorflow/contrib/tensor_forest/python/ops/gen_model_ops.py': No space left on device
/nix/store/np6ng5ap0r60zaiqpv7pl8l8i47ba25y-python2.7-tensorflow-1.11.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: grpc, pythonPackages.grpcio, python3Packages.grpcio, pythonPackages.grpcio-tools, python3Packages.grpcio-tools, pythonPackages.protobuf, python3Packages.protobuf

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/mib89j4gc0j6cb5ry57v3642qf9m9w50-python3.7-grpcio-tools-1.16.1/lib
patching script interpreter paths in /nix/store/mib89j4gc0j6cb5ry57v3642qf9m9w50-python3.7-grpcio-tools-1.16.1
checking for references to /build in /nix/store/mib89j4gc0j6cb5ry57v3642qf9m9w50-python3.7-grpcio-tools-1.16.1...
/nix/store/ywinq40p0i8lb4j857i3pa7m4x2k22zi-grpc-1.16.1
/nix/store/gjmsjwhda1qv5jrqzqa330z4h7sbhm46-python2.7-grpcio-1.16.1
/nix/store/xslyblgy6wf6ys03bzqld2i2p9fsbvd5-python3.7-grpcio-1.16.1
/nix/store/bg89p4vqyxxy3lwff4rk2j037wh7hfxr-python2.7-grpcio-tools-1.16.1
/nix/store/mib89j4gc0j6cb5ry57v3642qf9m9w50-python3.7-grpcio-tools-1.16.1
/nix/store/6lnxinmqa4qq0zk7lwzbn8nqk4xa39bf-python2.7-protobuf-3.6.1
/nix/store/pr232w15imyv2srnghg0m4alxp6plss5-python3.7-protobuf-3.6.1

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: grpc, pythonPackages.grpcio, python3Packages.grpcio, pythonPackages.grpcio-tools, python3Packages.grpcio-tools, pythonPackages.protobuf, python3Packages.protobuf

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/psbg3nr8kc2p66bcggw2ay4hrzhiznbk-python3.7-grpcio-tools-1.16.1/lib
patching script interpreter paths in /nix/store/psbg3nr8kc2p66bcggw2ay4hrzhiznbk-python3.7-grpcio-tools-1.16.1
checking for references to /build in /nix/store/psbg3nr8kc2p66bcggw2ay4hrzhiznbk-python3.7-grpcio-tools-1.16.1...
/nix/store/3jyvd3khr5sgjz9jba61l100qfcqc73x-grpc-1.16.1
/nix/store/q70b9scljyx8gjp9vvv20snwc1d9lygg-python2.7-grpcio-1.16.1
/nix/store/lfx2hjl33ckl1ph2p412ivi9a2m5bmpz-python3.7-grpcio-1.16.1
/nix/store/sq2hdf0kk2xay1cr14av2f0874k2wjar-python2.7-grpcio-tools-1.16.1
/nix/store/psbg3nr8kc2p66bcggw2ay4hrzhiznbk-python3.7-grpcio-tools-1.16.1
/nix/store/p73z6wzy4wgc51iql8s68avk1xnpyl61-python2.7-protobuf-3.6.1
/nix/store/25lsdslj7i6a327cxifnzswfii46lmfr-python3.7-protobuf-3.6.1

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: pythonPackages.tensorflow

Partial log (click to expand)

unpacking sources
patching sources
updateAutotoolsGnuConfigScriptsPhase
configuring
building
installing
/build/dist /build
tensorflow-1.11.0-cp27-none-linux_x86_64.whl is not a supported wheel on this platform.
builder for '/nix/store/1gx2jphzq10qx65364hmj1n06bvh3r4b-python2.7-tensorflow-1.11.0.drv' failed with exit code 1
error: build of '/nix/store/1gx2jphzq10qx65364hmj1n06bvh3r4b-python2.7-tensorflow-1.11.0.drv' failed

@worldofpeace
Copy link
Contributor

Seems pythonPackages.tensorflow has been broken on aarch64-linux for a while now https://hydra.nixos.org/job/nixpkgs/trunk/python27Packages.tensorflow.aarch64-linux/all

Copy link
Contributor

@worldofpeace worldofpeace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All usual checks pass for me. Thanks for picking this up 👍

@worldofpeace worldofpeace merged commit be8569b into NixOS:master Nov 27, 2018
@samueldr
Copy link
Member

The protobuf patch introduced here does not apply when using another (older) protobuf (e.g. protobuf_3_1).

$ nix-build -E 'with import ./. { }; python3Packages.protobuf.override { protobuf = protobuf3_1; }'
unpacking sources
unpacking source archive /nix/store/1zdyl0cxaa8ha2v1zp75zzdjd6j99d0m-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/util/python/BUILD
patching sources
applying patch /nix/store/yagx7hvylnnjq7lxbcia0y5lq1r736w3-0a59054c30e4f0ba10f10acfc1d7f3814c63e1a7.patch
patching file google/protobuf/pyext/descriptor.cc
Hunk #1 succeeded at 55 (offset -1 lines).
patching file google/protobuf/pyext/descriptor_containers.cc
patching file google/protobuf/pyext/descriptor_pool.cc
Hunk #1 succeeded at 47 (offset -1 lines).
patching file google/protobuf/pyext/extension_dict.cc
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED -- saving rejects to file google/protobuf/pyext/extension_dict.cc.rej
patching file google/protobuf/pyext/message.cc
Hunk #1 succeeded at 82 (offset 3 lines).
Hunk #2 succeeded at 1425 (offset -104 lines).

(PR incoming)

@lopsided98 lopsided98 deleted the grpc-update branch December 10, 2018 03:20
samueldr added a commit to samueldr/nixpkgs that referenced this pull request Dec 10, 2018
Change introduced in NixOS#50672.

The patch does not apply on older protobuf like protobuf3_1

```
$ nix-build -E 'with import ./. { }; python3Packages.protobuf.override { protobuf = protobuf3_1; }'
unpacking sources
unpacking source archive /nix/store/1zdyl0cxaa8ha2v1zp75zzdjd6j99d0m-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/util/python/BUILD
patching sources
applying patch /nix/store/yagx7hvylnnjq7lxbcia0y5lq1r736w3-0a59054c30e4f0ba10f10acfc1d7f3814c63e1a7.patch
patching file google/protobuf/pyext/descriptor.cc
Hunk #1 succeeded at 55 (offset -1 lines).
patching file google/protobuf/pyext/descriptor_containers.cc
patching file google/protobuf/pyext/descriptor_pool.cc
Hunk #1 succeeded at 47 (offset -1 lines).
patching file google/protobuf/pyext/extension_dict.cc
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED -- saving rejects to file google/protobuf/pyext/extension_dict.cc.rej
patching file google/protobuf/pyext/message.cc
Hunk #1 succeeded at 82 (offset 3 lines).
Hunk #2 succeeded at 1425 (offset -104 lines).
```

Since the patch isn't necessary on python versions older than 3.7, let's
only apply it for version 3.7. This means that most things using older
protobuf implementation will now be able to build when using an older
pythonPackage set (as is most probably the case anyway).

This still leaves protobuf 3.1 using packages hanging, but the errors
will be localized to those that would be breaking anyway with the
upgrade to 3.7 as default python.
dotlambda pushed a commit to samueldr/nixpkgs that referenced this pull request Dec 11, 2018
…g with it

Change introduced in NixOS#50672.

The patch does not apply on older protobuf like protobuf3_1

```
$ nix-build -E 'with import ./. { }; python3Packages.protobuf.override { protobuf = protobuf3_1; }'
unpacking sources
unpacking source archive /nix/store/1zdyl0cxaa8ha2v1zp75zzdjd6j99d0m-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/util/python/BUILD
patching sources
applying patch /nix/store/yagx7hvylnnjq7lxbcia0y5lq1r736w3-0a59054c30e4f0ba10f10acfc1d7f3814c63e1a7.patch
patching file google/protobuf/pyext/descriptor.cc
Hunk #1 succeeded at 55 (offset -1 lines).
patching file google/protobuf/pyext/descriptor_containers.cc
patching file google/protobuf/pyext/descriptor_pool.cc
Hunk #1 succeeded at 47 (offset -1 lines).
patching file google/protobuf/pyext/extension_dict.cc
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED -- saving rejects to file google/protobuf/pyext/extension_dict.cc.rej
patching file google/protobuf/pyext/message.cc
Hunk #1 succeeded at 82 (offset 3 lines).
Hunk #2 succeeded at 1425 (offset -104 lines).
```

Since the patch isn't necessary on python versions older than 3.7, let's
only apply it for version 3.7. This means that most things using older
protobuf implementation will now be able to build when using an older
pythonPackage set (as is most probably the case anyway).

This still leaves protobuf 3.1 using packages hanging, but the errors
will be localized to those that would be breaking anyway with the
upgrade to 3.7 as default python.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants