Should I set the maintainer to myself, to @domenkozar who maintains the pypy27 package I mostly copied, or both?

To yourself. @domenkozar no longer maintains the package actually. I'll update the maintainers in pkgs/interpreters/python/.

Should we make an effort to share common packaging code between pypy27 and pypy3?

Give it the same layout/structure/attributes, just as is done with the CPython 3 interpreters and mostly also with CPython 2.7.

How should we resolve the inconsistency between pypy27’s libPrefix = "pypy5.10" vs. pypy3’s libPrefix = "pypy3-5.10"? Should we change the former to "pypy-5.10" or "pypy27-5.10" or "pypy2.7-5.10"?

I think we should use respectively pypy2.7 and pypy3.5. It is not the PyPy version that matters, but the Python language version that is implemented.

Which of the isPy* flags should be set by pypy3? Should we set none of the existing ones and make a new isPypy3 flag? Or should we redefine isPypy for use by both pypy27 and pypy3, and have pypy27 set {isPy2, isPy27, isPypy} while pypy3 sets {isPy3, isPy35, isPypy}?

isPyPy is to indicate we have a PyPy interpreter. Which version of the language it implements does not matter. For that, we have e.g isPy35.

@GrahamcOfBorg build pypy3

Thanks @FRidh, I think I’ve addressed your feedback.

Failure for system: x86_64-darwin

Package ‘pypy3-5.10.1’ in […]/pypy/3/default.nix:137 is not supported on ‘x86_64-darwin’, refusing to evaluate.

I dunno if this would actually work on Darwin, but platforms = platforms.linux was copied from pypy27.

Failure for system: aarch64-linux

PyPy doesn’t support aarch64 (https://bitbucket.org/pypy/pypy/issues/2331/armv8-aarch64-or-aarch32-support)—do I need to do something here, and if so, how did pypy27 get away with this?

Failure for system: x86_64-linux

building of ‘/nix/store/ydrsminz1q71mvkr1qdh9warkr2kv3dr-pypy3-5.10.1.drv’ timed out after 3600 seconds

PyPy really does take a long time to build. We could build pypy3 using pypy27, which would be faster but create a longer build dependency chain?

While at it:

sitePackages = "lib/${libPrefix}/site-packages";

@FRidh Changed, but I think something’s wrong here, or at least incomplete? Before:

$ NIX_PATH="nixpkgs=$HOME/nixpkgs" nix-shell --pure -p 'pypy.withPackages (ps: [ps.setuptools])' --run 'pypy -c "import setuptools"'
Traceback (most recent call last):
  File "<module>", line 1, in <module>
ImportError: No module named setuptools

After:

$ NIX_PATH="nixpkgs=$HOME/nixpkgs" nix-shell --pure -p 'pypy.withPackages (ps: [ps.setuptools])' --run 'pypy -m setuptools'
these derivations will be built:
  /nix/store/nd18yl4wldiqbd0wx00wzn3k6b78138v-pypy2.7-setuptools-38.4.0.drv
  /nix/store/ar45rxvkch12hbcwf2imfs0891yblfjk-pypy-5.10.0-env.drv
building path(s) ‘/nix/store/a30dxjrbb8j04aq80cknk6m8x8nv1p5p-pypy2.7-setuptools-38.4.0’
[…]
Checking .pth file support in /nix/store/a30dxjrbb8j04aq80cknk6m8x8nv1p5p-pypy2.7-setuptools-38.4.0/site-packages/
/nix/store/j8ad3byf8pb8niqnwf38bf6q6rzb64ww-pypy-5.10.0/bin/pypy -E -c pass
TEST FAILED: /nix/store/a30dxjrbb8j04aq80cknk6m8x8nv1p5p-pypy2.7-setuptools-38.4.0/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /nix/store/a30dxjrbb8j04aq80cknk6m8x8nv1p5p-pypy2.7-setuptools-38.4.0/site-packages/

and your PYTHONPATH environment variable currently contains:

    '/nix/store/a30dxjrbb8j04aq80cknk6m8x8nv1p5p-pypy2.7-setuptools-38.4.0/lib/pypy2.7/site-packages:/nix/store/j8ad3byf8pb8niqnwf38bf6q6rzb64ww-pypy-5.10.0/lib/pypy2.7/site-packages:/nix/store/j8ad3byf8pb8niqnwf38bf6q6rzb64ww-pypy-5.10.0/lib/pypy2.7/site-packages'

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.
builder for ‘/nix/store/nd18yl4wldiqbd0wx00wzn3k6b78138v-pypy2.7-setuptools-38.4.0.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/ar45rxvkch12hbcwf2imfs0891yblfjk-pypy-5.10.0-env.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/ar45rxvkch12hbcwf2imfs0891yblfjk-pypy-5.10.0-env.drv’ failed
/run/current-system/sw/bin/nix-shell: failed to build all dependencies

Any progress?

This got stalled by a bug in the existing pypy package when I tried to test @FRidh’s suggestion of sitePackages = "lib/${libPrefix}/site-packages". I’ve split off that bug as #39356, and reverted the suggested change, as it seems to make things worse.

Give it the same layout/structure/attributes, just as is done with the CPython 3 interpreters and mostly also with CPython 2.7.

@FRidh why is it preferable to do this instead of using argument, as it has been done in chromium channels [1]? I found the diff size to be too small to warrant separate files.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/browsers/chromium/default.nix

The pypy2 changes previously part of this PR have been merged as #40459. So now this is a straightforward addition of pypy3 as a minimally modified copy of pypy2. Can we merge it?

(I could rework it so that common packaging code between pypy2 and pypy3 is shared, but was previously directed not to, so I’m leaving it this way unless otherwise instructed.)

In virtualenv we see the same structure on non-nixos btw.

pypy:

ls -la venv/
drwxr-xr-x  6 root root  7 Dec 20 12:31 .
drwx------  4 root root  6 Dec 20 12:31 ..
drwxr-xr-x  2 root root 20 Dec 20 12:31 bin
lrwxrwxrwx  1 root root 18 Dec 20 12:31 include -> /usr/local/include
drwxr-xr-x  3 root root  5 Dec 20 12:31 lib_pypy
drwxr-xr-x  3 root root  3 Dec 20 12:31 lib-python
drwxr-xr-x 10 root root 11 Dec 20 12:31 site-packages

python3:

ls -la venv/lib/python3.7/site-packages/
total 83
drwxr-xr-x 13 root root  14 Dec 20 12:35 .
drwxr-xr-x  8 root root  60 Dec 20 12:35 ..
drwxr-xr-x  2 root root   3 Dec 20 12:35 __pycache__
drwxr-xr-x  2 root root   5 Dec 20 12:35 bin
-rw-r--r--  1 root root 126 Dec 20 12:35 easy_install.py
drwxr-xr-x  2 root root   3 Dec 20 12:35 include
drwxr-xr-x  3 root root   3 Dec 20 12:35 lib
drwxr-xr-x  5 root root   7 Dec 20 12:35 pip
drwxr-xr-x  2 root root   9 Dec 20 12:35 pip-18.1.dist-info
drwxr-xr-x  5 root root   7 Dec 20 12:35 pkg_resources
drwxr-xr-x  6 root root  43 Dec 20 12:35 setuptools
drwxr-xr-x  2 root root  11 Dec 20 12:35 setuptools-40.6.3.dist-info
drwxr-xr-x  4 root root  12 Dec 20 12:35 wheel
drwxr-xr-x  2 root root   9 Dec 20 12:35 wheel-0.32.3.dist-info

we need to take this into account in our python infrastructure.

This should fix it: #52568