Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.setuptools: Remove windows files and make reproducible #71101

Closed
wants to merge 2 commits into from
Closed

pythonPackages.setuptools: Remove windows files and make reproducible #71101

wants to merge 2 commits into from

Conversation

singron
Copy link
Contributor

@singron singron commented Oct 14, 2019

Motivation for this change
  • setuptools includes *.exe files by default, but can be excluded with an ENV variable.
  • setuptools was built as an egg, which had reproducibility problems. Instead use a wheel

These are 2 separate commits, but I'm making one pull request against staging to limit rebuilds. Let me know if you would rather split it apart.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @FRidh

@singron
pythonPackages.setuptools: exclude windows files
38858b7 
These are various *.exe and *.xml files used only on windows. setuptools
includes them by default since it normally creates a single release for
all operating systems.

This reduces the size from 1020.0K to 801.6K according to `nix-path -sh`.
@singron
pythonPackages.setuptools: transition from egg to wheel
a606385 
The egg is a zip file. setuptools leaves timestamps in the egg,
which makes the build unreproducible. Unfortunately the files aren't
compressed so the size of setuptools increases to 2.3M from 0.8M
according to `nix path-info -sh`.

With this change, setuptools is reproducible according to

    nix-build -A python37Packages.setuptools --check
@singron singron requested a review from FRidh as a code owner October 14, 2019 02:51
@FRidh FRidh self-assigned this Oct 14, 2019
@FRidh
Copy link
Member

FRidh commented Oct 14, 2019

Thank you! I've pushed 90be4c2 instead to reuse the code that is in the hooks.

@FRidh FRidh closed this Oct 14, 2019
@timokau
Copy link
Member

timokau commented Oct 22, 2019

90be4c2 broke the python2.pkgs.twisted package. Reverting it fixes the build. Here's the error:

@nix { "action": "setPhase", "phase": "installPhase" }
installing
Executing pipInstallPhase
/build/Twisted-18.9.0/dist /build/Twisted-18.9.0
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python >
Processing ./Twisted-18.9.0-cp27-cp27mu-linux_x86_64.whl
Requirement already satisfied: Automat>=0.3.0 in /nix/store/11m5qcsi7wrxn910frk7330m5v9iwz1q-python2.7-Automat-0.7.0/lib/python2.7/site-packages (from Twisted==18.9.0) (0.7.0)
Requirement already satisfied: incremental>=16.10.1 in /nix/store/3v8r3c2yxgw6hhr095sjpn6ni7aivihf-python2.7-incremental-17.5.0/lib/python2.7/site-packages (from Twisted==18.9.0) (17.5.0)
Requirement already satisfied: zope.interface>=4.4.2 in /nix/store/ckj1nf5v6r76da62p73dlrgzi9azaj64-python2.7-zope.interface-4.6.0/lib/python2.7/site-packages (from Twisted==18.9.0) (4.6.0)
Requirement already satisfied: attrs>=17.4.0 in /nix/store/gi5sddwicyv1z4nplpxkdxdyz0a8vr9m-python2.7-attrs-18.2.0/lib/python2.7/site-packages (from Twisted==18.9.0) (18.2.0)
Requirement already satisfied: hyperlink>=17.1.1 in /nix/store/1px2m054rrsn4hf11js1hrr20d2ac4aq-python2.7-hyperlink-18.0.0/lib/python2.7/site-packages (from Twisted==18.9.0) (18.0.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in /nix/store/i1jdq72z5gs36w70slwc955z68cigdyf-python2.7-PyHamcrest-1.9.0/lib/python2.7/site-packages (from Twisted==18.9.0) (1.9.0)
Requirement already satisfied: constantly>=15.1 in /nix/store/fmn41vl2khs9l9fi2bngyxlga3i4vdaj-python2.7-constantly-15.1.0/lib/python2.7/site-packages (from Twisted==18.9.0) (15.1.0)
Requirement already satisfied: six in /nix/store/c2xxvfvnbg8z6pqnjx34mdhkrvzsdpbf-python2.7-six-1.12.0/lib/python2.7/site-packages (from Automat>=0.3.0->Twisted==18.9.0) (1.12.0)
Requirement already satisfied: setuptools in /nix/store/m3s2hmgrdnqgizjffbg5bjsxkj7jyhzp-python2.7-setuptools-41.2.0/lib/python2.7/site-packages (from zope.interface>=4.4.2->Twisted==18.9.0) (41.2.0)
Requirement already satisfied: idna>=2.5 in /nix/store/4zjn9dq1jhb7zk7ilrpj76fjinj5jy4a-python2.7-idna-2.8/lib/python2.7/site-packages (from hyperlink>=17.1.1->Twisted==18.9.0) (2.8)
Installing collected packages: Twisted
  WARNING: The scripts cftp, ckeygen, conch, mailmail, pyhtmlizer, tkconch, trial, twist and twistd are installed in '/nix/store/kzf182i4m4jq32qrj6ygfdgj7fky2frx-python2.7-Twisted-18.9.0/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Twisted-18.9.0
/build/Twisted-18.9.0
Traceback (most recent call last):
  File "/nix/store/kzf182i4m4jq32qrj6ygfdgj7fky2frx-python2.7-Twisted-18.9.0/bin/twistd", line 6, in <module>
    from twisted.scripts.twistd import run
  File "/nix/store/kzf182i4m4jq32qrj6ygfdgj7fky2frx-python2.7-Twisted-18.9.0/lib/python2.7/site-packages/twisted/scripts/twistd.py", line 13, in <module>
    from twisted.application import app
  File "/nix/store/kzf182i4m4jq32qrj6ygfdgj7fky2frx-python2.7-Twisted-18.9.0/lib/python2.7/site-packages/twisted/application/app.py", line 17, in <module>
    from twisted import copyright, plugin, logger
  File "/nix/store/kzf182i4m4jq32qrj6ygfdgj7fky2frx-python2.7-Twisted-18.9.0/lib/python2.7/site-packages/twisted/plugin.py", line 18, in <module>
    from zope.interface import Interface, providedBy
ImportError: No module named zope.interface

@timokau
Copy link
Member

timokau commented Oct 22, 2019 •
edited

This likely has to do with twisted being called in postInstall:

  # Generate Twisted's plug-in cache.  Twisted users must do it as well.  See
  # http://twistedmatrix.com/documents/current/core/howto/plugin.html#auto3
  # and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477103 for
  # details.
  postInstall = "$out/bin/twistd --help > /dev/null";

@FRidh FRidh mentioned this pull request Oct 23, 2019
Closed
@FRidh
Copy link
Member

FRidh commented Oct 23, 2019

This is indeed the cause of the regression. Thanks for bisecting! Note I did not push this commit, but pushed 90be4c2 instead. It uses hooks instead of the phases defined here, however, it eventually runs the same commands.

The next question is, what part exactly is causing it. So, it is not the environment variable SETUPTOOLS_INSTALL_WINDOWS_SPECIFIC_FILES but it is the change from egg to wheel. When building as egg, the twisted build succeeds, when building setuptools as wheel, it does not.

@FRidh
Copy link
Member

FRidh commented Oct 23, 2019

Fix in #71792.

@marzipankaiser
Copy link
Contributor

This breaks webtest, see #71824

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

Assignees

@FRidh FRidh

Labels
6.topic: python 6.topic: reproducible builds 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@singron @FRidh @timokau @marzipankaiser