Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.mypy: compile with mypyc #105462

Merged
merged 1 commit into from Dec 27, 2020
Merged

python3Packages.mypy: compile with mypyc #105462

merged 1 commit into from Dec 27, 2020

Conversation

ruuda
Copy link
Contributor

@ruuda ruuda commented Nov 30, 2020 •
edited

Motivation for this change

Mypy includes mypyc, a compiler that translates annotated Python to C, which can be compiled into a Python module. When mypy itself is compiled with mypyc, it is about 4 times faster than the interpreted mypy, so using the compiled version is especially useful on large codebases where typechecking can take a long time.

The wheels distributed on Pypi have included a mypyc-compiled mypy by default since version 0.700, but because those wheels are platform-specific, we can't take advantage of them. With this change, we build mypy with mypyc from source, so it should work across all supported platforms.

Ideally taking advantage of mypyc would be as simple as setting MYPY_USE_MYPYC=1, but due to a few issues that have since been fixed upstream, this is not possible for v0.790. I expect that in a future version that will work, but for now, take the source from GitHub (because the package on Pypi is missing some files), and apply a few patches.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Running the bare mypyc binary fails with this message:

Traceback (most recent call last):
  File "build/setup.py", line 2, in <module>
    from mypyc.build import mypycify
ModuleNotFoundError: No module named 'mypyc'

But it also does that on master, so that is not a change introduced by this PR.

You can tell that mypyc did its work because lib/python3.8/site-packages/mypy is now full of .cpython-38-x86_64-linux-gnu.so-files for every module. These are the compiled modules.

@ruuda
python3Packages.mypy: compile with mypyc
ad26cb9 
Mypy includes mypyc, a compiler that translates annotated Python to C,
which can be compiled into a Python module. When mypy is compiled with
mypyc, it is about 4 times faster than the interpreted mypy, so using
the compiled version is especially useful on large codebases where
typechecking may take a long time.

The wheels distributed on Pypi have included a mypyc-compiled mypy by
default since version 0.700 [1].

[1]: http://mypy-lang.blogspot.com/2019/04/mypy-0700-released-up-to-4x-faster.html
@ruuda
Copy link
Contributor Author

ruuda commented Nov 30, 2020

To illustrate the advantage, here is the perf stat for mypy from master checking a moderately sized Python codebase:

 Performance counter stats for 'result/bin/mypy «redacted»':

     217502.555807      task-clock (msec)         #    1.000 CPUs utilized          
               704      context-switches          #    0.003 K/sec                  
                 0      cpu-migrations            #    0.000 K/sec                  
           171,363      page-faults               #    0.788 K/sec                  
 1,080,701,702,560      cycles                    #    4.969 GHz                    
 1,662,325,108,477      instructions              #    1.54  insn per cycle         
   379,739,736,485      branches                  # 1745.909 M/sec                  
     7,531,505,307      branch-misses             #    1.98% of all branches        

     217.594915343 seconds time elapsed

And here for this PR:

 Performance counter stats for 'result/bin/mypy «redacted»':

      42234.077128      task-clock (msec)         #    1.000 CPUs utilized          
                60      context-switches          #    0.001 K/sec                  
                 0      cpu-migrations            #    0.000 K/sec                  
           146,279      page-faults               #    0.003 M/sec                  
   208,946,235,716      cycles                    #    4.947 GHz                    
   205,834,101,324      instructions              #    0.99  insn per cycle         
    55,392,870,039      branches                  # 1311.568 M/sec                  
       627,015,962      branch-misses             #    1.13% of all branches        

      42.235955573 seconds time elapsed

@ofborg ofborg bot requested review from martingms and LnL7 November 30, 2020 16:39
@SuperSandro2000
Copy link
Member

Result of nixpkgs-review pr 105462 run on x86_64-linux 1

2 packages marked as broken and skipped:
  • bareos
  • pepper
1 package blacklisted:
  • tests.nixos-functions.nixosTest-test
54 packages failed to build:
  • ceph (ceph-dev)
  • ceph-client
  • coconut (python38Packages.coconut)
  • errbot
  • fava
  • flexget
  • hydra-check (python38Packages.hydra-check)
  • libceph
  • mnemosyne
  • mypy (python38Packages.mypy)
  • nix-pin
  • python37Packages.WSME
  • python37Packages.algebraic-data-types
  • python37Packages.cheroot
  • python37Packages.cherrypy
  • python37Packages.coconut
  • python37Packages.hydra-check
  • python37Packages.ics
  • python37Packages.irc
  • python37Packages.jaraco_logging
  • python37Packages.mypy
  • python37Packages.portend
  • python37Packages.pyalgotrade
  • python37Packages.pyls-mypy
  • python37Packages.pynamodb
  • python37Packages.pytest-mypy
  • python37Packages.tatsu
  • python37Packages.tempora
  • python37Packages.typer
  • python37Packages.ws4py
  • python37Packages.zerobin
  • python38Packages.WSME
  • python38Packages.algebraic-data-types
  • python38Packages.cheroot
  • python38Packages.cherrypy
  • python38Packages.ics
  • python38Packages.irc
  • python38Packages.jaraco_logging
  • python38Packages.portend
  • python38Packages.pyalgotrade
  • python38Packages.pyls-mypy
  • python38Packages.pynamodb
  • python38Packages.pytest-mypy
  • python38Packages.tatsu
  • python38Packages.tempora
  • python38Packages.typer
  • python38Packages.ws4py
  • python38Packages.zerobin
  • sabnzbd
  • sambaFull (samba4Full)
  • thonny
  • tribler
  • ulauncher
  • zfs-replicate

@SuperSandro2000 SuperSandro2000 merged commit 05140d2 into NixOS:master Dec 27, 2020
@SuperSandro2000
Copy link
Member

Result of nixpkgs-review pr 105462 run on x86_64-darwin 1

2 packages marked as broken and skipped:
  • bareos
  • pepper
2 packages failed to build:
  • mnemosyne
  • python38Packages.pyls-mypy
43 packages built:
  • coconut (python38Packages.coconut)
  • fava
  • flexget
  • hydra-check (python38Packages.hydra-check)
  • mypy (python38Packages.mypy)
  • nix-pin
  • python37Packages.WSME
  • python37Packages.algebraic-data-types
  • python37Packages.cheroot
  • python37Packages.cherrypy
  • python37Packages.coconut
  • python37Packages.hydra-check
  • python37Packages.ics
  • python37Packages.irc
  • python37Packages.jaraco_logging
  • python37Packages.mypy
  • python37Packages.portend
  • python37Packages.pyalgotrade
  • python37Packages.pyls-mypy
  • python37Packages.pynamodb
  • python37Packages.pytest-mypy
  • python37Packages.tatsu
  • python37Packages.tempora
  • python37Packages.typer
  • python37Packages.ws4py
  • python37Packages.zerobin
  • python38Packages.WSME
  • python38Packages.algebraic-data-types
  • python38Packages.cheroot
  • python38Packages.cherrypy
  • python38Packages.ics
  • python38Packages.irc
  • python38Packages.jaraco_logging
  • python38Packages.portend
  • python38Packages.pyalgotrade
  • python38Packages.pynamodb
  • python38Packages.pytest-mypy
  • python38Packages.tatsu
  • python38Packages.tempora
  • python38Packages.typer
  • python38Packages.ws4py
  • python38Packages.zerobin
  • zfs-replicate

vcunat added a commit to vcunat/nixpkgs that referenced this pull request Dec 30, 2020
@vcunat
python3Packages.mypy: don't use mypyc on 32-bit platforms
3e40d6d 
Almost all i686-linux tests got blocked because of this problem:
  https://hydra.nixos.org/eval/1638038#tabs-now-fail
It regressed in PR NixOS#105462 (commit ad26cb9).

Now I tested that at least some test got fixed:
  nix build -f nixos/release-combined.nix nixos.tests.knot.i686-linux
This change won't even cause any rebuild on 64-bit platforms,
and using nix booleans seems nicer anyway.
@vcunat vcunat mentioned this pull request Dec 30, 2020
Merged
10 tasks
FRidh pushed a commit that referenced this pull request Dec 30, 2020
@vcunat @FRidh
python3Packages.mypy: don't use mypyc on 32-bit platforms
95f685d 
Almost all i686-linux tests got blocked because of this problem:
  https://hydra.nixos.org/eval/1638038#tabs-now-fail
It regressed in PR #105462 (commit ad26cb9).

Now I tested that at least some test got fixed:
  nix build -f nixos/release-combined.nix nixos.tests.knot.i686-linux
This change won't even cause any rebuild on 64-bit platforms,
and using nix booleans seems nicer anyway.
@ruuda ruuda deleted the mypy-mypyc branch October 6, 2022 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SuperSandro2000 SuperSandro2000 SuperSandro2000 left review comments

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

@martingms martingms Awaiting requested review from martingms

@LnL7 LnL7 Awaiting requested review from LnL7

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ruuda @SuperSandro2000