Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python: enable optimizations #43442

Closed
wants to merge 1 commit into from
Closed

python: enable optimizations #43442

wants to merge 1 commit into from

Conversation

LnL7
Copy link
Member

@LnL7 LnL7 commented Jul 12, 2018

Motivation for this change

I didn't perform any tests yet, but I assume it's pretty safe and hopefully makes some stuff a bit faster.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
      • python27
      • python34
      • python35
      • python36
      • python37
    • macOS
      • python27
      • python34
      • python35
      • python36
      • python37
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@LnL7 LnL7 requested a review from FRidh as a code owner July 12, 2018 22:15
@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python

Partial log (click to expand)

cannot build derivation '/nix/store/q52d46p2pn1n7j9sdlj026p2sw8qh1wf-python-2.7.15.drv': 14 dependencies couldn't be built
cannot build derivation '/nix/store/51l8zz3492nhd81crmywfrxw8dxiy31h-libtool-2.4.6.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/fzzcy71yps97c1zbgmn9ghd8zy0aqvwj-hook.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/49g5wc9ici5hiw5dkx7a5rsbdzy5il90-flex-2.6.4.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/bh7kcpgdbqh0qnd59wkg4ab9nw1whvkp-bootstrap_cmds-dev-tools-7.0.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/3zkaa0hlii6rl4y7kms68m8zrglrfmr4-xnu-osx-10.11.6.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/dq2lbvnjwdvl5nzc6c6qp59wk1rc6bfr-IOKit-osx-10.11.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/jymfkn4gv2bx1kfysy33v29wqir1iahv-configd-osx-10.8.5.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/p9jaqmm84rw418vywa6j1zdkbw7wf1vl-python-2.7.15.drv': 15 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/p9jaqmm84rw418vywa6j1zdkbw7wf1vl-python-2.7.15.drv' failed

@orivej
Copy link
Contributor

orivej commented Jul 12, 2018

This should go to staging.

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: python

Partial log (click to expand)

gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/import.o Python/import.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -I. -o Python/importdl.o ./Python/importdl.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/marshal.o Python/marshal.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/modsupport.o Python/modsupport.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/mystrtoul.o Python/mystrtoul.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/peephole.o Python/peephole.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-use -fprofile-correction  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/pyarena.o Python/pyarena.c
building of ‘/nix/store/p1s33b4jvc727yyaxifryzm6nqrvccs7-python-2.7.15.drv’ timed out after 1800 seconds
error: build of ‘/nix/store/p1s33b4jvc727yyaxifryzm6nqrvccs7-python-2.7.15.drv’ failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python

Partial log (click to expand)

0:32:32 load avg: 81.04 [188/399] test_isinstance -- test_ioctl skipped
0:32:34 load avg: 82.56 [189/399] test_iter
0:32:38 load avg: 82.56 [190/399] test_iterlen
0:32:39 load avg: 89.08 [191/399] test_itertools
0:36:18 load avg: 95.85 [192/399] test_json -- test_itertools passed in 3 min 39 sec
0:37:50 load avg: 95.16 [193/399] test_kqueue -- test_json passed in 1 min 31 sec
0:37:51 load avg: 95.16 [194/399] test_largefile -- test_kqueue skipped
0:37:52 load avg: 95.16 [195/399] test_lib2to3
building of '/nix/store/sjyjccjvxcqr1kf6ij509j4jx9i28294-python-2.7.15.drv' timed out after 3600 seconds
error: build of '/nix/store/sjyjccjvxcqr1kf6ij509j4jx9i28294-python-2.7.15.drv' failed

@andir
Copy link
Member

andir commented Jul 13, 2018

Have you considered enabling LTO as well? https://hg.python.org/cpython/file/3.6/configure.ac#l1287

@FRidh
Copy link
Member

FRidh commented Jul 13, 2018

We would have to test against the package set to see if it does not have any unwanted side-effects.

@FRidh
Copy link
Member

FRidh commented Jul 13, 2018

How much more time does it take to build the different Python versions?

@LnL7 LnL7 changed the base branch from master to staging July 14, 2018 14:33
@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python

Partial log (click to expand)

cannot build derivation '/nix/store/s2amfy3krl19l5xs2jlbvcmkzk371y37-python-2.7.15.drv': 14 dependencies couldn't be built
cannot build derivation '/nix/store/0706xvwvs6fi4l4r19vg70b3in3z56fw-libtool-2.4.6.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ab0gb1d08pkppvlylvmpgc6nv1wsawbh-hook.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/pnh5qavvgzla65kj3yspg9an315sjm49-flex-2.6.4.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/y9bl1l8s94s3jsa0ncz8d53gf6dkq3hr-bootstrap_cmds-dev-tools-7.0.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/64n4zsr0yrzlx64qg0gkajsji55n9j1b-xnu-osx-10.11.6.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/jidrjk5j8k0m1y4114arvkbwvf5x56gp-IOKit-osx-10.11.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/xypq024ba02npfdzxbhg4m6fj4bbnxm9-configd-osx-10.8.5.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/l54w00xqk3m0hvb2blkwi8hzyzxbxi08-python-2.7.15.drv': 15 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/l54w00xqk3m0hvb2blkwi8hzyzxbxi08-python-2.7.15.drv' failed

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: python

Partial log (click to expand)

gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_EXPAT_CONFIG_H=1 -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/./Modules/expat -I. -IInclude -I./Include -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Include -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15 -c /tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/_elementtree.c -o build/temp.linux-x86_64-2.7/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/_elementtree.o
gcc -pthread -shared -lgcc_s -lgcc_s -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fprofile-use -fprofile-correction build/temp.linux-x86_64-2.7/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/_elementtree.o -L. -lpython2.7 -o build/lib.linux-x86_64-2.7/_elementtree.so
building '_multibytecodec' extension
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Include -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15 -c /tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/cjkcodecs/multibytecodec.c -o build/temp.linux-x86_64-2.7/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/cjkcodecs/multibytecodec.o
gcc -pthread -shared -lgcc_s -lgcc_s -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fprofile-use -fprofile-correction build/temp.linux-x86_64-2.7/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/cjkcodecs/multibytecodec.o -L. -lpython2.7 -o build/lib.linux-x86_64-2.7/_multibytecodec.so
building '_codecs_kr' extension
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Include -I/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15 -c /tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/cjkcodecs/_codecs_kr.c -o build/temp.linux-x86_64-2.7/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/cjkcodecs/_codecs_kr.o
gcc -pthread -shared -lgcc_s -lgcc_s -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fprofile-use -fprofile-correction build/temp.linux-x86_64-2.7/tmp/nix-build-python-2.7.15.drv-0/Python-2.7.15/Modules/cjkcodecs/_codecs_kr.o -L. -lpython2.7 -o build/lib.linux-x86_64-2.7/_codecs_kr.so
building of ‘/nix/store/mb1drg9p8s5pj6x7wnn1mgg53p2k0gx7-python-2.7.15.drv’ timed out after 1800 seconds
error: build of ‘/nix/store/mb1drg9p8s5pj6x7wnn1mgg53p2k0gx7-python-2.7.15.drv’ failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python

Partial log (click to expand)

0:51:15 load avg: 4.05 [229/399] test_nntplib
0:51:16 load avg: 4.05 [230/399] test_normalization
0:51:17 load avg: 4.05 [231/399] test_ntpath
0:51:19 load avg: 4.05 [232/399] test_old_mailbox
0:51:20 load avg: 3.96 [233/399] test_openpty
0:51:21 load avg: 3.96 [234/399] test_operator
0:51:22 load avg: 3.96 [235/399] test_optparse
0:51:25 load avg: 3.89 [236/399] test_ordered_dict
building of '/nix/store/hlg12sic6d2wb05zg4bnwr3w8j1mbxb7-python-2.7.15.drv' timed out after 3600 seconds
error: build of '/nix/store/hlg12sic6d2wb05zg4bnwr3w8j1mbxb7-python-2.7.15.drv' failed

@LnL7
Copy link
Member Author

LnL7 commented Jul 15, 2018

@FRidh I tested this with the dependency tree of a project as a sanity check and didn't notice any problems.
The build time does indeed increase significantly because of the tests, but it's not like it was long already. Shouldn't we be running some of the tests anyway?

-nix-build -A python27 --check  0.17s user 0.08s system 0% cpu 1:39.95 total
-nix-build -A python36 --check  0.19s user 0.10s system 0% cpu 1:34.19 total
+nix-build -A python27 --check  0.19s user 0.09s system 0% cpu 15:40.69 total
+nix-build -A python36 --check  0.23s user 0.12s system 0% cpu 28:26.26 total

@vcunat
Copy link
Member

vcunat commented Aug 30, 2018

I wonder... do you care for this for 18.09? (I can't really afford to review it myself.)

@LnL7
Copy link
Member Author

LnL7 commented Aug 30, 2018

I reread the description of the flag and I'm not actually sure anymore if it's useful. I mainly tried this out because the build log recommends it for a release build.

If you want a release build with all optimizations active (LTO, PGO, etc),
please run ./configure --enable-optimizations

@FRidh
Copy link
Member

FRidh commented Aug 31, 2018

@LnL7 https://bugs.python.org/msg281332

@lukateras
Copy link
Member

lukateras commented Oct 13, 2018 •
edited

https://bugs.python.org/msg281332

It seems we're good on that since --with-lto is not passed and --enable-optimizations doesn't imply LTO since 2016-09.

Will merge in a few days if there's no disagreement.

@lukateras lukateras self-assigned this Oct 13, 2018
@dtzWill
Copy link
Member

dtzWill commented Oct 19, 2018

So I imagine it's not as much of a problem in practice but I can't help but be a little uncertain about PGO'ing python using a test-suite that experiences failures (which I'm seeing on at least 3.7 and probably others). Note that apparently the tests failing is not a build-breaker, but it does make one wonder.... what exactly are optimizing towards? 🙁

On a related note, anyone know what sort of performance benefits this gives (on whatever metrics you like)? Is it known to cause regressions in certain areas? (Presumably not many of importance since. as you say, this flag is recommended when not specified...)

This is all about optimizing the python interpreter itself, yes? Are there similar options / capabilities for optimizing (or pre-compiling bytecode at all) python packages/modules as well?

@FRidh
Copy link
Member

FRidh commented Oct 20, 2018

At least when building the interpreter we also build optimized bytecode. We could do that as well in a hook for all Python packages.

I don't think we should bother with these optimizations. If you indeed have something critical running, then likely you know about the option and enable it.

@erictapen erictapen mentioned this pull request Feb 5, 2019
Merged
@FRidh FRidh closed this May 12, 2019
@FRidh FRidh mentioned this pull request Jul 23, 2019
Closed
10 tasks
@sbourdeauducq
Copy link
Contributor

sbourdeauducq commented Jan 4, 2020 •
edited

If you indeed have something critical running, then likely you know about the option and enable it.

This is a rather difficult proposal since it means rebuilding a lot of packages that depend on Python.
What about enabling --enable-optimizations and --with-lto only on certain platforms, like x86_64-linux, where toolchains are better tested, and CPU power to build Python is relatively plentiful and low-cost?

FWIW I've just run nix-build -A python27 -A python35 -A python36 -A python37 -A python38 -A python39 on x86_64-linux with nixpkgs master and those flags; the build finished without issue and the resulting python binaries ran fine.

@sbourdeauducq sbourdeauducq mentioned this pull request Jan 4, 2020
Closed
@FRidh FRidh mentioned this pull request Apr 2, 2020
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matthewbauer matthewbauer matthewbauer approved these changes

@lukateras lukateras lukateras approved these changes

@FRidh FRidh Awaiting requested review from FRidh FRidh is a code owner

Assignees

@lukateras lukateras

Labels
6.topic: python 10.rebuild-darwin: 501+ 10.rebuild-darwin-stdenv 10.rebuild-linux: 501+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
10 participants
@LnL7 @GrahamcOfBorg @orivej @andir @FRidh @vcunat @lukateras @dtzWill @sbourdeauducq @matthewbauer