pythonPackages.rlp: 1.0.1 -> 1.0.2 (with some new deps) #45644

jluttine · 2018-08-26T11:20:31Z

Motivation for this change

Updated and fixed RLP Python package. Some dependencies are added as separate commits.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

FRidh · 2018-08-26T11:45:01Z

It's nowadays possible to use markdown directly, so no need for setuptools_markdown.
https://packaging.python.org/guides/making-a-pypi-friendly-readme/
A more recent setuptools and wheel are needed though.

Could you inform upstream?

jluttine · 2018-08-26T12:09:10Z

Ok. I noticed nox-review is giving some failures at the moment. I'll try to fix them before merging.

jluttine · 2018-08-26T12:52:36Z

@FRidh Now nox-review passed. I needed to add some more commits to make the fixes.

Mic92 · 2018-08-27T07:38:36Z

pkgs/development/python-modules/backports_shutil_which/default.nix

+  doCheck = false;
+
+  meta = with stdenv.lib; {
+    description = "Backport of shutil.which from Python 3.3";


Maybe python 3 should be disabled here?

should be fine: https://github.com/minrk/backports.shutil_which/blob/master/backports/shutil_which.py#L75

Mic92 · 2018-08-27T07:43:14Z

pkgs/development/python-modules/backports_shutil_which/default.nix

+    sha256 = "16sa3adkf71862cb9pk747pw80a2f1v5m915ijb4fgj309xrlhyx";
+  };
+
+  # Tests fail: "ValueError: underlying buffer has been detached"


Is this comment copy-and-pasted from python-u2flib-host?

I tried to build and the test phase raised that error. I tried to fix it but it has probably something to do with the test environment in nix. Not sure though.

For me this test works perfectly fine.

Mic92 · 2018-08-27T07:47:19Z

Upstream is also informed: ethereum/eth-utils#124

Mic92 · 2018-08-27T09:36:44Z

@GrahamcOfBorg build python3Packages.backports_shutil_which

GrahamcOfBorg · 2018-08-27T09:37:18Z

Success on aarch64-linux (full log)

Attempted: python3Packages.backports_shutil_which

Partial log (click to expand)

running install tests
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /build/backports.shutil_which-3.5.1, inifile:
collected 2 items

test/test_which.py ..                                                    [100%]

=========================== 2 passed in 0.06 seconds ===========================
/nix/store/xzxa0jvmxbbnbpivwb89cm5ihkpqrrg6-python3.6-backports.shutil_which-3.5.1

GrahamcOfBorg · 2018-08-27T09:37:22Z

Success on x86_64-darwin (full log)

Attempted: python3Packages.backports_shutil_which

Partial log (click to expand)

running install tests
============================= test session starts ==============================
platform darwin -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /private/tmp/nix-build-python3.6-backports.shutil_which-3.5.1.drv-0/backports.shutil_which-3.5.1, inifile:
collected 2 items

test/test_which.py ..                                                    [100%]

=========================== 2 passed in 0.12 seconds ===========================
/nix/store/rqnij3zfp85735yrfiixcwrxjmq2wi7w-python3.6-backports.shutil_which-3.5.1

GrahamcOfBorg · 2018-08-27T09:38:11Z

Success on x86_64-linux (full log)

Attempted: python3Packages.backports_shutil_which

Partial log (click to expand)

running install tests
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /build/backports.shutil_which-3.5.1, inifile:
collected 2 items

test/test_which.py ..                                                    [100%]

=========================== 2 passed in 0.06 seconds ===========================
/nix/store/qpdyvfbqdsj62fwq55vjnim3441c6blg-python3.6-backports.shutil_which-3.5.1

jluttine · 2018-08-29T09:19:18Z

Is this ok now?

dotlambda · 2018-08-29T15:12:26Z

pkgs/development/python-modules/PyMsgBox/default.nix

+
+  meta = with stdenv.lib; {
+    description = "A simple, cross-platform, pure Python module for JavaScript-like message boxes";
+    homepage = "https://github.com/asweigart/PyMsgBox";


Homepage says this depends on tkinter. Btw, you should leave out the quotes.

dotlambda · 2018-08-29T15:13:02Z

pkgs/development/python-modules/eth-hash/default.nix

+    substituteInPlace setup.py --replace \'setuptools-markdown\' ""
+  '';
+
+  disabled = !(pythonAtLeast "3.5");


pythonOlder

dotlambda · 2018-08-29T15:13:21Z

pkgs/development/python-modules/eth-typing/default.nix

+    substituteInPlace setup.py --replace \'setuptools-markdown\' ""
+  '';
+
+  disabled = !(pythonAtLeast "3.5");


pythonOlder

dotlambda · 2018-08-29T15:15:20Z

pkgs/top-level/python-packages.nix

@@ -437,6 +439,8 @@ in {

  pymatgen = callPackage ../development/python-modules/pymatgen { };

+  PyMsgBox = callPackage ../development/python-modules/PyMsgBox { };


I'd prefer using pymsgbox, also as dir name.

Yes, I can change the dir name. How about adding below that:

pymsgbox = PyMsgBox;

I've heard the package name should be equal to the name in PyPI, so that's why I thought it should be available at least as PyMsgBox.

I would also prefer if it matches. We should have a policy on that, so we do this consistently for python packages.

How about a policy to have both the PyPI name and a lowercase alias?

I've written this before several times. We prefer to use normalized names throughout, because that matches well with the naming we use throughout Nixpkgs. The pname to fetchPypi however has to match to the filename used because PyPI does not translate that part of the url. If for fetchPypi the name cannot be normalized, then it may be more convenient to use the same one for buildPython*.

Upstream issue: https://github.com/pypa/warehouse/issues/2537

Ok. let's make this convention part of nixpkgs doc: #45822 (comment)

So I should rename backports_shutil_which to backports-shutil-which, right?

GrahamcOfBorg · 2018-09-11T12:27:48Z

Success on x86_64-darwin (full log)

Attempted: python3.pkgs.eth-hash, python3.pkgs.eth-typing, python3.pkgs.eth-utils

The following builds were skipped because they don't evaluate on x86_64-darwin: pythob2.pkgs.eth-hash, python2.pkgs.eth-typing, python2.pkgs.eth-utils

Partial log (click to expand)

while evaluating anonymous function at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/python-modules/eth-hash/default.nix:1:1, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/customisation.nix:69:12:
while evaluating 'makeOverridablePythonPackage' at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/top-level/python-packages.nix:40:37, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/python-modules/eth-hash/default.nix:3:1:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/top-level/python-packages.nix:42:12:
while evaluating anonymous function at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/interpreters/python/build-python-package.nix:27:1, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/lib/customisation.nix:69:12:
while evaluating anonymous function at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/interpreters/python/mk-python-derivation.nix:15:1, called from /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/development/interpreters/python/build-python-package.nix:44:4:
eth-hash-0.1.4 not supported for interpreter python2.7

/nix/store/sxh6s4d9pghxp8jr3qzybfsqrh37ba4q-python3.6-eth-hash-0.1.4
/nix/store/5q0a18fy1kkm7hh8hnv3kmp9jfkfy3qr-python3.6-eth-typing-1.1.0
/nix/store/vh7di2czkif07fnycf6iqi3ir2sv14is-python3.6-eth-utils-1.2.0

GrahamcOfBorg · 2018-09-11T12:28:13Z

Success on aarch64-linux (full log)

Attempted: python3.pkgs.eth-hash, python3.pkgs.eth-typing, python3.pkgs.eth-utils

The following builds were skipped because they don't evaluate on aarch64-linux: pythob2.pkgs.eth-hash, python2.pkgs.eth-typing, python2.pkgs.eth-utils

Partial log (click to expand)

while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/development/python-modules/eth-hash/default.nix:1:1, called from /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/lib/customisation.nix:69:12:
while evaluating 'makeOverridablePythonPackage' at /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/top-level/python-packages.nix:40:37, called from /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/development/python-modules/eth-hash/default.nix:3:1:
while evaluating 'makeOverridable' at /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/lib/customisation.nix:67:24, called from /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/top-level/python-packages.nix:42:12:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/development/interpreters/python/build-python-package.nix:27:1, called from /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/lib/customisation.nix:69:12:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/development/interpreters/python/mk-python-derivation.nix:15:1, called from /var/lib/gc-of-borg/nix-test-rs-29/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-29/pkgs/development/interpreters/python/build-python-package.nix:44:4:
eth-hash-0.1.4 not supported for interpreter python2.7

/nix/store/y0pyiz25vg9sah0kl4nw2xgq4n6zs755-python3.6-eth-hash-0.1.4
/nix/store/jawr54kbpx7radxd1q1vvsphqbrngypx-python3.6-eth-typing-1.1.0
/nix/store/rs6lbj4nwp47fppzk77pxay3p6qvx7ym-python3.6-eth-utils-1.2.0

GrahamcOfBorg · 2018-09-11T12:28:43Z

Success on x86_64-linux (full log)

Attempted: python3.pkgs.eth-hash, python3.pkgs.eth-typing, python3.pkgs.eth-utils

The following builds were skipped because they don't evaluate on x86_64-linux: pythob2.pkgs.eth-hash, python2.pkgs.eth-typing, python2.pkgs.eth-utils

Partial log (click to expand)

eth-typing==1.1.0
eth-utils==1.2.0
pycrypto==3.6.6
pycryptodome==3.6.6
pysha3==1.0.2
toolz==0.9.0

/nix/store/qav2n7z0bw34ggl2iar8525ipjdq6vcx-python3.6-eth-hash-0.1.4
/nix/store/pxvzfjvk8y6y3h9mdlci0akw6livds63-python3.6-eth-typing-1.1.0
/nix/store/0s3ixsbj1zzvvbqppghr6gqz059350pc-python3.6-eth-utils-1.2.0

dotlambda · 2018-09-11T12:30:20Z

All of the eth-* packages give

Ran 0 tests in 0.000s

You should either provide an appropriate checkPhase or set doCheck = false and add a comment with the reason.
Otherwise, this looks good to go.

jluttine · 2018-09-11T12:39:26Z

At least eth-utils doesn't include tests in the PyPI release tarball. I'll report it upstream.

dotlambda · 2018-09-11T12:43:13Z

Thanks! It's fine to set doCheck = false for now but don't forget to make a PR here once they include them in the PyPI tarball.

jluttine · 2018-09-11T15:11:45Z

I put doCheck = false to eth-utils and eth-typing as they don't have unit tests in the PyPI release tarball. I reported these upstream: ethereum/eth-utils#130 and ethereum/eth-typing#8.

I enabled tests for eth-hash, but that required some tweaking.

samueldr · 2018-09-12T01:20:17Z

Once ported to 18.09, it would contribute to fixing around 12 outstanding builds on x86_64, see ZHF #45960, and the generated reports. (Subscribing to keep track of the changes.)

jluttine · 2018-09-12T05:32:02Z

Upstream of eth-utils and eth-typing doesn't want to include tests in PyPI release tarball so I'll modify the package definitions to use GitHub sources so they can be tested.

jluttine · 2018-09-12T05:49:49Z

Done. I also updated the version numbers of eth-utils and eth-typing. Nox review passed.

dotlambda · 2018-09-12T10:56:08Z

Thanks a lot @jluttine!
Would you mind opening a PR against release-18.09 or do you want me to do that?

jluttine · 2018-09-12T11:20:17Z

How does that work? Shall I cherry pick these 10 commits to that release branch? I can do it but I've never done it before so need to learn it.

dotlambda · 2018-09-12T11:24:51Z

Well, then it's a good thing to learn :)
You have to use git cherry-pick -x on the release-18.09 branch. Also see https://nixos.org/nixpkgs/manual/#idm140737315686368.
And then, make sure to select release-18.09 when making the PR.
Btw, I think git cherry-pick accepts commit ranges.

samueldr · 2018-09-13T02:16:25Z

$ git cherry-pick -x c44504410ba5a3bd76a99e7907533779c2b17694^..8d4afa5f2ef0fbdf042069ac71479bba1
104bbc8

[release-18.09 735b1e1] pythonPackages.eth-hash: init at 0.1.14
Date: Sun Aug 26 14:09:30 2018 +0300
2 files changed, 47 insertions(+)
create mode 100644 pkgs/development/python-modules/eth-hash/default.nix
[release-18.09 923a6b2] pythonPackages.eth-typing: init at 1.1.0
Date: Sun Aug 26 14:11:02 2018 +0300
2 files changed, 37 insertions(+)
create mode 100644 pkgs/development/python-modules/eth-typing/default.nix
[release-18.09 d72f81e] pythonPackages.eth-utils: init at 1.2.0
Date: Sun Aug 26 14:12:06 2018 +0300
2 files changed, 37 insertions(+)
create mode 100644 pkgs/development/python-modules/eth-utils/default.nix
[release-18.09 832dcfd] pythonPackages.rlp: fix build
Date: Sun Aug 26 14:13:24 2018 +0300
1 file changed, 13 insertions(+), 3 deletions(-)
[release-18.09 f8c8a23] pythonPackages.python-u2flib-host: init at 3.0.3
Date: Sun Aug 26 15:15:27 2018 +0300
2 files changed, 25 insertions(+)
create mode 100644 pkgs/development/python-modules/python-u2flib-host/default.nix
[release-18.09 ff6e4d0] pythonPackages.ledgerblue: fix build inputs
Date: Sun Aug 26 15:16:08 2018 +0300
1 file changed, 7 insertions(+), 2 deletions(-)
[release-18.09 825f91b] pythonPackage.pymsgbox: init at 1.0.6
Date: Sun Aug 26 15:43:42 2018 +0300
2 files changed, 26 insertions(+)
create mode 100644 pkgs/development/python-modules/pymsgbox/default.nix
[release-18.09 91f488c] pythonPackage.backports-shutil-which: init at 3.5.1
Date: Sun Aug 26 15:44:16 2018 +0300
2 files changed, 23 insertions(+)
create mode 100644 pkgs/development/python-modules/backports-shutil-which/default.nix
[release-18.09 ed0fc7d] pythonPackages.libagent: fix propagatedBuildInputs
Date: Sun Aug 26 15:44:38 2018 +0300
1 file changed, 5 insertions(+), 9 deletions(-)
[release-18.09 29f4541] pythonPackage.backports-shutil-which: enable tests
Date: Mon Aug 27 10:35:54 2018 +0100
1 file changed, 6 insertions(+), 3 deletions(-)

Pushed for ZHF #45960

jluttine requested a review from FRidh as a code owner August 26, 2018 11:20

jluttine force-pushed the update-rlp-1.0.2 branch 2 times, most recently from 30c4ccd to 4728055 Compare August 26, 2018 11:24

GrahamcOfBorg added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Aug 26, 2018

jluttine force-pushed the update-rlp-1.0.2 branch from 4728055 to 61eb081 Compare August 26, 2018 11:46

GrahamcOfBorg added 8.has: clean-up 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 11-100 labels Aug 26, 2018

GrahamcOfBorg added 10.rebuild-linux: 11-100 and removed 10.rebuild-linux: 1-10 labels Aug 26, 2018

Mic92 reviewed Aug 27, 2018

View reviewed changes

dotlambda reviewed Aug 29, 2018

View reviewed changes

Mic92 mentioned this pull request Aug 30, 2018

doc/python: convention for attributes names #45822

Merged

9 tasks

nyanloutre mentioned this pull request Sep 5, 2018

electrum and electron-cash: add support for btchip library #40816

Merged

8 tasks

pythonPackages.eth-hash: init at 0.1.14

c445044

jluttine force-pushed the update-rlp-1.0.2 branch from 9202945 to d44bb56 Compare September 11, 2018 14:57

jluttine added 7 commits September 12, 2018 08:44

pythonPackages.eth-typing: init at 1.1.0

541a13c

pythonPackages.eth-utils: init at 1.2.0

9f8e614

pythonPackages.rlp: fix build

f053daf

pythonPackages.python-u2flib-host: init at 3.0.3

38e86a8

pythonPackages.ledgerblue: fix build inputs

5c9db2f

pythonPackage.pymsgbox: init at 1.0.6

c9bf7f6

pythonPackage.backports-shutil-which: init at 3.5.1

80b64a2

jluttine force-pushed the update-rlp-1.0.2 branch from d44bb56 to fea2d86 Compare September 12, 2018 05:44

jluttine and others added 2 commits September 12, 2018 09:04

pythonPackages.libagent: fix propagatedBuildInputs

d0ce883

pythonPackage.backports-shutil-which: enable tests

8d4afa5

jluttine force-pushed the update-rlp-1.0.2 branch from fea2d86 to 8d4afa5 Compare September 12, 2018 06:04

dotlambda merged commit c15a94b into NixOS:master Sep 12, 2018

xeji mentioned this pull request Sep 12, 2018

18.09 Zero Hydra Failures #45960

Closed

samueldr removed the 9.needs: port to stable A PR needs a backport to the stable release. label Apr 17, 2019

		@@ -437,6 +439,8 @@ in {

		pymatgen = callPackage ../development/python-modules/pymatgen { };

		PyMsgBox = callPackage ../development/python-modules/PyMsgBox { };

pythonPackages.rlp: 1.0.1 -> 1.0.2 (with some new deps) #45644

pythonPackages.rlp: 1.0.1 -> 1.0.2 (with some new deps) #45644

Conversation

jluttine commented Aug 26, 2018 • edited by dotlambda

Motivation for this change

Things done

FRidh commented Aug 26, 2018

jluttine commented Aug 26, 2018

jluttine commented Aug 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jluttine Aug 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Aug 27, 2018

Mic92 commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

jluttine commented Aug 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Aug 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

dotlambda commented Sep 11, 2018 • edited

jluttine commented Sep 11, 2018

dotlambda commented Sep 11, 2018

jluttine commented Sep 11, 2018

samueldr commented Sep 12, 2018

jluttine commented Sep 12, 2018

jluttine commented Sep 12, 2018

dotlambda commented Sep 12, 2018

jluttine commented Sep 12, 2018

dotlambda commented Sep 12, 2018 • edited

samueldr commented Sep 13, 2018 • edited

jluttine commented Aug 26, 2018 •

edited by dotlambda

jluttine Aug 27, 2018 •

edited

Mic92 Aug 30, 2018 •

edited

dotlambda commented Sep 11, 2018 •

edited

dotlambda commented Sep 12, 2018 •

edited

samueldr commented Sep 13, 2018 •

edited