Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[reverted] python2 and python3: reproducible builds #107965

Merged
merged 1 commit into from Mar 13, 2021
Merged

[reverted] python2 and python3: reproducible builds #107965

merged 1 commit into from Mar 13, 2021

Conversation

FRidh
Copy link
Member

@FRidh FRidh commented Dec 30, 2020 •
edited

Achieve reproducible builds of the interpreter. Note this meant
disabling optimizations again that were introduced in #84072.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

todo:

  • changelog
  • fix comments

@FRidh FRidh requested a review from raboof December 30, 2020 13:50
@FRidh
Copy link
Member Author

FRidh commented Dec 30, 2020

After testing I did some reordering resulting in a mass rebuild. I have not tested this new commit (its ongoing).

@FRidh FRidh mentioned this pull request Dec 30, 2020
Closed
@FRidh
Copy link
Member Author

FRidh commented Dec 30, 2020 •
edited

Non-optimized bytecode is now removed, which I think means __debug__ = false.

I don't know which .pyc is actually picked up when bytecode is available for multiple optimization levels.

@FRidh FRidh changed the base branch from master to staging December 30, 2020 15:01
FRidh
FRidh commented Dec 30, 2020
View reviewed changes
pkgs/development/interpreters/python/cpython/2.7/default.nix Outdated Show resolved Hide resolved
pkgs/development/interpreters/python/cpython/2.7/default.nix
find $out -name "*.py" | $out/bin/python -m compileall -q -f -x "lib2to3" -i -
find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x "lib2to3" -i -
find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x "lib2to3" -i -
find $out -name "*.py" | ${pythonForBuildInterpreter} -O -m compileall -q -f -x "lib2to3" -i -
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python 2 stores optimized bytecode in .pyo and stores only one level, not two. Thus we need to choose.

pkgs/development/interpreters/python/cpython/2.7/default.nix Outdated Show resolved Hide resolved
@FRidh FRidh changed the title python3: reproducible builds python2 and python3: reproducible builds Dec 30, 2020
@FRidh FRidh mentioned this pull request Dec 30, 2020
Merged
16 tasks
@prusnak prusnak linked an issue Dec 30, 2020 that may be closed by this pull request
python3 library build is not deterministic #107951
Closed
raboof
raboof reviewed Jan 4, 2021
View reviewed changes
Copy link
Member

@raboof raboof left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thrilled this indeed seems to make python reproducible. On the other hand, I can see how disabling the optimizations is painful (not sure about the stripping)...

@drewrisinger
Copy link
Contributor

I looked through this PR briefly, and I was wondering if it's possible to have the optimizations as a deterministic input. That would probably be a separate PR, though.

Something like:

  1. Generate optimizations as part of a (new) derivation.
  2. Use optimizations as input to the python interpreter build. This could either be an optional or default argument, I'm open to either.
  3. Output a deterministic, but optimized, python interpreter.

Honestly, I haven't looked enough into the python optimization process to know if this is possible, it just seemed like a best-of-both-worlds scenario.

@FRidh FRidh added this to the 21.05 milestone Jan 25, 2021
@tomberek
Copy link
Contributor

Result of nixpkgs-review pr 107965 run on x86_64-linux 1

1 package built:
  • python3

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-21-05/11559/4

@FRidh
Copy link
Member Author

FRidh commented Feb 22, 2021

I will go ahead with this soon, at least before 21.05. Need to make some minor changes, and add release notes and docs.

@raboof raboof self-requested a review February 22, 2021 15:19
@FRidh FRidh mentioned this pull request Feb 23, 2021
Merged
10 tasks
@FRidh
Copy link
Member Author

FRidh commented Mar 10, 2021

  • comment needs to be fixed
  • documentation needs to be updated
  • release notes should be added

@FRidh FRidh force-pushed the python3-reproducible branch 2 times, most recently from 1aba2fb to 915a30f Compare March 13, 2021 12:02
@FRidh
python: reproducible builds
8883f85 
Achieve reproducible builds of the interpreter. Note this meant
disabling optimizations again.
@FRidh FRidh merged commit 9d03ff5 into NixOS:staging Mar 13, 2021
@FRidh FRidh deleted the python3-reproducible branch March 13, 2021 12:11
@FRidh FRidh added this to In progress in R13y via automation Mar 13, 2021
@FRidh FRidh moved this from In progress to Done in R13y Mar 13, 2021
@FRidh FRidh moved this from Done to In progress in R13y Mar 13, 2021
@raboof raboof mentioned this pull request Apr 28, 2021
Merged
10 tasks
@drewrisinger drewrisinger mentioned this pull request Jul 2, 2021
Closed
@Artturin Artturin moved this from In progress to Done in R13y Oct 16, 2021
@Artturin
Copy link
Member

@FRidh why is this in in progress even though its merged?

@FRidh
Copy link
Member Author

FRidh commented Oct 16, 2021

Because this has been reverted.

@Artturin Artturin moved this from Done to In progress in R13y Oct 16, 2021
@Artturin Artturin changed the title python2 and python3: reproducible builds [reverted] python2 and python3: reproducible builds Oct 16, 2021
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drewrisinger drewrisinger Awaiting requested review from drewrisinger

@raboof raboof Awaiting requested review from raboof

Assignees
No one assigned
Labels
6.topic: nixos 6.topic: python 6.topic: reproducible builds 8.has: changelog 8.has: documentation 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-darwin-stdenv 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ 10.rebuild-linux-stdenv 11.by: package-maintainer
Projects
R13y
In progress
Milestone
21.05
Development

Successfully merging this pull request may close these issues.

python3 library build is not deterministic
7 participants
@FRidh @drewrisinger @tomberek @nixos-discourse @Artturin @raboof @SuperSandro2000