pythonPackages.pex: 1.2.7 -> 1.4.5 #45497

mgeisler · 2018-08-23T09:48:16Z

Motivation for this change

This packages a new version of pex and moves it to a top-level tool, similarly to how Pipenv is packaged.

I'm not sure what the impact is of the move from pkgs/top-level/python-packages.nix to pkgs/top-level/all-packages.nix, but since pex is more of a command line tool than a Python library, the change makes sense to me :-) Please advice if this is bad for some reason or if there is followup work to be done!

The packaging was tested locally on a MacOS machine and a Ubuntu VM. The packaging was also executed on a Hydra installation with the sandbox enabled.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Ubuntu
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

mgeisler · 2018-08-23T10:03:38Z

undefined variable 'pex' at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/eval-2-shlevy.ewr1.nix.ci/pkgs/development/tools/build-managers/pants/default.nix:28:61

Okay... I guess this is a hint that moving pex does have an impact :-)

Mic92 · 2018-08-23T17:21:52Z

Do you know if it is used as an executable or as a library in pantsbuild? This is were the evaluation error is currently coming from.

dotlambda · 2018-08-24T07:49:50Z

pkgs/development/tools/pex/default.nix

@@ -0,0 +1,29 @@
+{ stdenv, python3Packages }:
+with python3Packages; buildPythonApplication rec {
+    name = "${pname}-${version}";


Drop this line

Okay, thanks -- is it computed automatically from pname and version?

dotlambda · 2018-08-24T07:51:01Z

pkgs/top-level/all-packages.nix

@@ -7620,6 +7620,7 @@ with pkgs;

  pew = callPackage ../development/tools/pew {};
  pipenv = callPackage ../development/tools/pipenv {};
+  pex = callPackage ../development/tools/pex {};


alphabetic order and blank lines

Thanks, I've inserted the package between pew and pipenv now.

dotlambda · 2018-08-24T07:51:34Z

pkgs/development/tools/pex/default.nix

+    patches = [ ./set-source-date-epoch.patch ];
+
+    # A few more dependencies I don't want to handle right now...
+    doCheck = false;


Did you check whether these dependencies have been packaged in the meantime?

No, I just copied the existing definition. Removing the line seems to work fine -- though no tests are run.

Please let me know how you like to see changes made in response to review comments: as follow-up commits, or by me immediately fixing the broken commit and force pushing?

mgeisler · 2018-08-24T14:46:22Z

Do you know if it is used as an executable or as a library in pantsbuild?

@Mic92, I looked it up and Pants do use PEX as a Python library.

So I guess that means that I should not be moving from buildPythonPackage to buildPythonApplication like I originally thought?

Mic92 · 2018-08-24T15:40:36Z

Yes otherwise pantsbuild might break because no dependencies from pex to pantsbuild are propagated: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix#L66

@FRidh would it make sense to have also a buildPythonApplication variant of the same package that does not propagate dependencies?

FRidh · 2018-08-25T11:28:38Z

I don't really see the value in having a buildPythonApplication variant. How does it deal with multiple interpreter versions?

mgeisler · 2018-08-27T09:05:01Z

I don't really see the value in having a buildPythonApplication variant.

Oh, I don't know if there's any value in it either :-) I just saw that Pipenv was packaged that way and since PEX is also a command line too, I thought that using buildPythonApplication would be a little more correct somehow.

I'll try updating it back to buildPythonPackage and see if that works better. Thanks for the tips!

mgeisler · 2018-08-27T09:53:00Z

pkgs/development/python-modules/pex/default.nix

+  propagatedBuildInputs = [ setuptools wheel requests ];
+
+  prePatch = ''
+    substituteInPlace setup.py --replace 'SETUPTOOLS_REQUIREMENT,' '"setuptools",'


I'm a little worried about this line -- which was part of the original derivation. The setup.py file from pex includes a pex/version.py file, which says:

# NB: If we upgrade to setuptools>=34 pex's bootstrap code in `PEXBuilder` will need an update to # include the `packaging` package in the `.bootstrap/` code since we use # `packaging.specifiers.SpecifierSet` - indirectly - through `pkg_resources.Requirement.specifier`. SETUPTOOLS_REQUIREMENT = 'setuptools>=20.3,<34.0' WHEEL_REQUIREMENT = 'wheel>=0.26.0,<0.32.0'

The SETUPTOOLS_REQUIREMENT and WHEEL_REQUIREMENT variables are then used in setup.py. When we change the requirement to just "setuptools", we get whatever version is in Nix -- version 40.0.0 at the moment. This sounds dangerous in light of the above comment.

Is there a normal approach in Nix to handle situations like this? I imagine one could package setuptools version 33 and have pex use that specific version?

With python we are more bound to having only a single version in nixpkgs because of dependency propagation. However it seems that master also support setuptools newer then 34: https://github.com/pantsbuild/pex/blob/f125317bb6001ccbb56f68168bc45f947d274405/pex/version.py#L8

Ah, cool! The new version.py file says:

# Versions 34.0.0 through 35.0.2 (last pre-36.0.0) de-vendored dependencies which # causes problems for pex code so we exclude that range. SETUPTOOLS_REQUIREMENT = 'setuptools>=20.3,<41,!=34.*,!=35.*'

which means that using version 40.0.0 should be fine. Thanks for looking this up!

Mic92 · 2018-08-27T12:45:10Z

@GrahamcOfBorg build pythonPackages.pex python3Packages.pex

GrahamcOfBorg · 2018-08-27T12:45:57Z

Success on x86_64-darwin (full log)

Attempted: pythonPackages.pex, python3Packages.pex

Partial log (click to expand)

warning: no previously-included files matching '*~' found under directory '*'
writing manifest file 'pex.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/izc6s44r6y1kpq5l1fs01f98s6b0x7db-python2.7-pex-1.4.5
/nix/store/cn7ismrgfib91vbpkqqx5y33l6m2h514-python3.6-pex-1.4.5

GrahamcOfBorg · 2018-08-27T12:46:04Z

Failure on aarch64-linux (full log)

Attempted: pythonPackages.pex, python3Packages.pex

Partial log (click to expand)

Reading https://pypi.org/simple/pytest/
Download error on https://pypi.org/simple/pytest/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'pytest' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for pytest
error: Could not find suitable distribution for Requirement.parse('pytest')
builder for '/nix/store/7j76m6x2l1r6942adavm7c6ry73q4fil-python3.6-pex-1.4.5.drv' failed with exit code 1
error: build of '/nix/store/7j76m6x2l1r6942adavm7c6ry73q4fil-python3.6-pex-1.4.5.drv', '/nix/store/kbvirbswfdhjjffmhx8dzmwf65cmwk8z-python2.7-pex-1.4.5.drv' failed

GrahamcOfBorg · 2018-08-27T12:46:39Z

Failure on x86_64-linux (full log)

Attempted: pythonPackages.pex, python3Packages.pex

Partial log (click to expand)

Reading https://pypi.org/simple/pytest/
Download error on https://pypi.org/simple/pytest/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'pytest' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for pytest
error: Could not find suitable distribution for Requirement.parse('pytest')
builder for '/nix/store/m4rnfxhryrd36ccsih9sqmd7hg1cn3f4-python3.6-pex-1.4.5.drv' failed with exit code 1
error: build of '/nix/store/24xkcyzzxnkj4d81404c5bqngw5dvlw0-python2.7-pex-1.4.5.drv', '/nix/store/m4rnfxhryrd36ccsih9sqmd7hg1cn3f4-python3.6-pex-1.4.5.drv' failed

Mic92 · 2018-08-27T12:47:39Z

pkgs/development/python-modules/pex/default.nix

+  };
+
+  propagatedBuildInputs = [ setuptools wheel requests ];
+


Blind guess:

checkInputs = [ pytest ];

to fix the build issues:

Thanks, I'll give it a try..

The install_requires have changed from version 1.2.7 to 1.4.5 and now also include the wheel package. So this and setuptools have been added to propagatedBuildInputs.

The files in /nix/store are all back-dated to 1970, and this breaks pex when it tries to generate wheels: wheels are really zip files, and the zip file format does not support dates before 1980. This is normally handled by setting SOURCE_DATE_EPOCH when building the Nix package. However, pex will invoke further Python interpreters at runtime in calls like this: $ python setup.py bdist_wheel We thus need to ensure that it passes the SOURCE_DATE_EPOCH variable onto those interpreters. The date of the wheels will be fixed at 1980.

Without requests, pex will emit a number of warnings to the terminal when building pex files: % ./result/bin/pex -v httpie -o http pex: Warning, using a UrllibContext which is known to be flaky. pex: Please build pex with the requests module for more reliable downloads. pex: Warning, using a UrllibContext which is known to be flaky. pex: Please build pex with the requests module for more reliable downloads. This fixes these warnings.

Mic92 · 2018-08-27T17:09:47Z

@GrahamcOfBorg build pythonPackages.pex python3Packages.pex

GrahamcOfBorg · 2018-08-27T17:10:28Z

Success on x86_64-darwin (full log)

Attempted: pythonPackages.pex, python3Packages.pex

Partial log (click to expand)

warning: no previously-included files matching '*~' found under directory '*'
writing manifest file 'pex.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/7hy485pmia4h6mhfp1fhv4aimxfs72a4-python2.7-pex-1.4.5
/nix/store/0318k83l6lbpznfpxh4qlcmd1p6p9g3l-python3.6-pex-1.4.5

GrahamcOfBorg · 2018-08-27T17:10:32Z

Failure on aarch64-linux (full log)

Attempted: pythonPackages.pex, python3Packages.pex

Partial log (click to expand)

Reading https://pypi.org/simple/twitter.common.dirutil/
Download error on https://pypi.org/simple/twitter.common.dirutil/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'twitter.common.dirutil' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for twitter.common.dirutil<0.4.0,>=0.3.1
error: Could not find suitable distribution for Requirement.parse('twitter.common.dirutil<0.4.0,>=0.3.1')
builder for '/nix/store/1afi9zw1hlbn4mhz81k5b01ybf4291c7-python3.6-pex-1.4.5.drv' failed with exit code 1
error: build of '/nix/store/1afi9zw1hlbn4mhz81k5b01ybf4291c7-python3.6-pex-1.4.5.drv', '/nix/store/6kfijr4ma8gif468arl5v6p13xnw7w8p-python2.7-pex-1.4.5.drv' failed

GrahamcOfBorg · 2018-08-27T17:11:12Z

Failure on x86_64-linux (full log)

Attempted: pythonPackages.pex, python3Packages.pex

Partial log (click to expand)

Reading https://pypi.org/simple/twitter.common.dirutil/
Download error on https://pypi.org/simple/twitter.common.dirutil/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'twitter.common.dirutil' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for twitter.common.dirutil<0.4.0,>=0.3.1
error: Could not find suitable distribution for Requirement.parse('twitter.common.dirutil<0.4.0,>=0.3.1')
builder for '/nix/store/7h3k3y90hywx49s8xdn89pkxqszhxgby-python3.6-pex-1.4.5.drv' failed with exit code 1
error: build of '/nix/store/7h3k3y90hywx49s8xdn89pkxqszhxgby-python3.6-pex-1.4.5.drv', '/nix/store/kw8hpc3pviqwlb0rwp68nw4sdw435qb5-python2.7-pex-1.4.5.drv' failed

Mic92 · 2018-08-27T17:25:23Z

we also have twitter-common-dirutil in nixpkgs.

Mic92 · 2018-08-28T07:45:29Z

You can reproduce these problems locally, when enabling nix's sandbox.

mgeisler · 2018-08-31T13:43:14Z

Thanks @Mic92 -- I'll have to experiment a bit more with this offline! You're completely right that I should get my local setup done so that I can find a way to reproduce what you see.

I'll ping you guys later when I have worked out the kinks... thanks for all the feedback until now!

mgeisler · 2019-01-14T16:05:09Z

Hi guys -- I no longer work with Nix, so I won't be able to finish this. I hope the PR can be useful to someone else who wants to upgrade PEX.

mgeisler requested a review from FRidh as a code owner August 23, 2018 09:48

GrahamcOfBorg added the 6.topic: python label Aug 23, 2018

dotlambda reviewed Aug 24, 2018

View reviewed changes

mgeisler force-pushed the pex-1.4.5 branch from fabcfcb to f3dc19a Compare August 24, 2018 14:33

mgeisler force-pushed the pex-1.4.5 branch from 235dcd9 to 606adfd Compare August 27, 2018 09:47

mgeisler commented Aug 27, 2018

View reviewed changes

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Aug 27, 2018

mgeisler changed the title ~~pex: 1.2.7 -> 1.4.5~~ pythonPackages.pex: 1.2.7 -> 1.4.5 Aug 27, 2018

Mic92 reviewed Aug 27, 2018

View reviewed changes

mgeisler force-pushed the pex-1.4.5 branch from 606adfd to ef2d7ba Compare August 27, 2018 14:16

mgeisler added 4 commits August 27, 2018 16:17

pex: move definition to python-modules/pex

16047f9

pex: 1.2.7 -> 1.4.5

eb9ee54

The install_requires have changed from version 1.2.7 to 1.4.5 and now also include the wheel package. So this and setuptools have been added to propagatedBuildInputs.

mgeisler force-pushed the pex-1.4.5 branch from ef2d7ba to fe28a84 Compare August 27, 2018 14:17

FRidh added the 2.status: work-in-progress label Sep 6, 2018

mgeisler closed this Jan 14, 2019

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.pex: 1.2.7 -> 1.4.5 #45497

pythonPackages.pex: 1.2.7 -> 1.4.5 #45497

mgeisler commented Aug 23, 2018

mgeisler commented Aug 23, 2018

Mic92 commented Aug 23, 2018 •

edited

dotlambda Aug 24, 2018

mgeisler Aug 24, 2018

dotlambda Aug 24, 2018

mgeisler Aug 24, 2018 •

edited

dotlambda Aug 24, 2018

mgeisler Aug 24, 2018

mgeisler commented Aug 24, 2018

Mic92 commented Aug 24, 2018 •

edited

FRidh commented Aug 25, 2018

mgeisler commented Aug 27, 2018

mgeisler Aug 27, 2018

Mic92 Aug 27, 2018 •

edited

mgeisler Aug 27, 2018

Mic92 commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

Mic92 Aug 27, 2018

mgeisler Aug 27, 2018

Mic92 commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

Mic92 commented Aug 27, 2018 •

edited

Mic92 commented Aug 28, 2018

mgeisler commented Aug 31, 2018

mgeisler commented Jan 14, 2019

pythonPackages.pex: 1.2.7 -> 1.4.5 #45497

pythonPackages.pex: 1.2.7 -> 1.4.5 #45497

Conversation

mgeisler commented Aug 23, 2018

Motivation for this change

Things done

mgeisler commented Aug 23, 2018

Mic92 commented Aug 23, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgeisler Aug 24, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgeisler commented Aug 24, 2018

Mic92 commented Aug 24, 2018 • edited

FRidh commented Aug 25, 2018

mgeisler commented Aug 27, 2018

Choose a reason for hiding this comment

Mic92 Aug 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

GrahamcOfBorg commented Aug 27, 2018

Mic92 commented Aug 27, 2018 • edited

Mic92 commented Aug 28, 2018

mgeisler commented Aug 31, 2018

mgeisler commented Jan 14, 2019

Mic92 commented Aug 23, 2018 •

edited

mgeisler Aug 24, 2018 •

edited

Mic92 commented Aug 24, 2018 •

edited

Mic92 Aug 27, 2018 •

edited

Mic92 commented Aug 27, 2018 •

edited