@veprbl I had to stop the python.pkgs.numpy from being propagated, as this would potentially export a different python interpreter than the hepmc3 python modules.

Thanks for the bump! I have a comment about numpy:

@veprbl I had to stop the python.pkgs.numpy from being propagated, as this would potentially export a different python interpreter than the hepmc3 python modules.

I assume you mean a different numpy package?

IIRC, ROOT links against the numpy library, so using it with any other numpy library could break. Python libraries in nixpkgs usually use propagatedBuildInputs for their dependency on numpy (or any other python library), but we are not changing those. Some other options include: removing numpy support or instantiating pythonPackages.root.

IIRC, ROOT links against the numpy library, so using it with any other numpy library could break. Python libraries in nixpkgs usually use propagatedBuildInputs for their dependency on numpy (or any other python library), but we are not changing those. Some other options include: removing numpy support or instantiating pythonPackages.root.

then we should probably use something like passthru to do this. As propagating python.pkgs.numpy will also introduce the python38 interpreter into the user's environment which is strange, if a user also wanted to use python37, but now has a python38 on his/her path.

cc @FRidh

As propagating python.pkgs.numpy will also introduce the python38 interpreter into the user's environment which is strange

Must admit, I did not know about this:

# nix-shell -p python37 -p python38Packages.numpy --pure --run "python3 --version"
Python 3.7.6
# nix-shell -p python38Packages.numpy --pure --run "python3 --version"
Python 3.8.2
# nix-shell -p python37Packages.numpy --pure --run "python3 --version"
Python 3.7.6
# nix-shell -p python37Packages.pytest --pure --run "python3 --version"
Python 3.7.6

I still don't see how this is a problem of this specific expression. If you change it, you might as well change it everywhere in pkgs/development/python-modules.

I altered hepmc3 to override the python interpreter passed to root, so they should be coherent. But not a very elegant solution

I still don't see how this is a problem of this specific expression. If you change it, you might as well change it everywhere in pkgs/development/python-modules.

because root is defined in all-packages.nix and hepmc3 is defined in python-modules. with exception of when they both get passed the same python version, it will be incoherent.

For example, the python37Packages.hepmc3 on master fails due to compilation issues, but with the bump it will fail because it's trying to install for a different python interpreter

[13:21:19] jon@nixos ~/projects/nixpkgs (master)
$ nix-build -A python37Packages.hepmc3
...
-- Found Python3: /nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/bin/python3.8 (found suitable version "3.8.3", minimum required is "3") found components:  Development Interpreter
-- HepMC3 python: Python verson 3.X found in /nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/bin/python3.8. Python bindings generation is possible.
-- HepMC3 python: Tweek HEPMC3_Python_SITEARCH38 option to set the installation path for the python38 bindings.
-- HepMC3 python: HEPMC3_Python_SITEARCH38 defaults to /nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/site-package
...

I also added an installcheckphase, just to make sure it's able to be imported

https://github.com/NixOS/nixpkgs/pull/92078
4 packages built:
fastnlo python27Packages.hepmc3 python37Packages.hepmc3 python38Packages.hepmc3

You remove the explicit python dependency, which means that you treat interpreter propagation as a useful feature that we should rely on, but then you remove pythonPackages.numpy from propagatedBuildInputs and motivate that by saying that the users will not expect interpreter to be propagated. I would really like some clarification on what is our stance on interpreter propagation before starting changing things back and forth.

I'm coming from the user's perspective. It's surprising to have the interpreter I didn't ask for in my nix-shell environment, but for the build environment, i feel like it's much less surprising to have the interpreter present.

If I had the following shell.nix:

with import <nixpkgs> {};

pkgs.mkShell {
  buildInputs = [
    python37
    root
  ];
}

I would expect to have python37 as my python interpreter, and for root to not bleed into my environment.

[15:50:52] jon@nixos ~/projects/nixpkgs (master)
$ nix-shell --pure -p "with import ./. {}; [ root python37 ]"

[nix-shell:~/projects/nixpkgs]$ python --version
Python 3.8.3
...
[15:51:20] jon@nixos ~/projects/nixpkgs (bump-hepmc3)
$ nix-shell --pure -p "with import ./. {}; [ root python37 ]"

[nix-shell:~/projects/nixpkgs]$ python --version
Python 3.7.7

I would really like some clarification on what is our stance on interpreter propagation before starting changing things back and forth.

You can never expect nix-shell -p python37Packages.numpy to work because Python packages don't necessarily propagate the interpreter. In a build it means explicitly adding python37 to buildInputsor, in case of nix-shell, adding the corresponding -p python37. Having said that, that's still an unsupported way of working.

About the propagation one can say it's bad either way. If we propagate, packages may get numpy when they don't want it (hence we should avoid propagation altogether), however, if we don't, we break Python packages that depend on this package requiring numpy. Clearly, this issue should be fixed in a proper way and it's on the todo list (the passthru idea with a custom propagation logic).

If I had the following shell.nix:
....
I would expect to have python37 as my python interpreter, and for root to not bleed into my environment.

That is something you cannot expect because nix-shell and mkShell run setup hooks, it's a tool for reproducing a build environment after all. I don't think we should be optimizing for that type of use, despite it being very common.

Now I don't know how these tools use numpy, but from what I understood I think it makes sense to ensure they both get build with the same interpreter version. But not propagating numpy is something I have a mixed opinion on.

That is something you cannot expect because nix-shell and mkShell run setup hooks, it's a tool for reproducing a build environment after all. I don't think we should be optimizing for that type of use, despite it being very common.

It was also causing a build failure for python37Packages.hepmc3 as the python38 interpreter was the one that was found.

Let me summarize my understanding of the situation.

The python interpreter propagation originally added 6 years ago in ab423e6 and its original purpose for introducing it was for convenience of use in nix-shell. It is currently implemented as:

As we can see, it affects only python packages defined using the buildPythonPackage, so, for example, python3Packages.numpy will propagate python3, but python3Packages.yoda will not. Hence:

You can never expect nix-shell -p python37Packages.numpy to work because Python packages don't necessarily propagate the interpreter. In a build it means explicitly adding python37 to buildInputsor, in case of nix-shell, adding the corresponding -p python37. Having said that, that's still an unsupported way of working.

Then, I think, the @jonringer 's original concern was that one does not expect a package referred to as root to propagate a specific python package version (unlike a hypothetical python3Packages.root which would at least clearly indicate a version).

About the propagation one can say it's bad either way. If we propagate, packages may get numpy when they don't want it (hence we should avoid propagation altogether), however, if we don't, we break Python packages that depend on this package requiring numpy.

I actually was not worried too much because if someone is using a ROOT API which returns a numpy array (like ROOT.RDataFrame(tree).AsNumpy()), they are very likely to do some sort of processing for it, at that point including the python3Packages.numpy explicitly in the environment becomes a reasonable thing to do. What I had in mind was that one wants to have the same numpy as was used by ROOT, but perhaps this would be better achieved by adding inherit (python.pkgs) numpy to the expression for ROOT (I suppose, this is what @jonringer was suggesting in #92078 (comment)) and the using it as buildInputs = [ root root.numpy ];.

Clearly, this issue should be fixed in a proper way and it's on the todo list (the passthru idea with a custom propagation logic).

I don't know, but I am curious what other kind of propagation we could do.

Overall it doesn't seem like the interpreter propagation feature is doing much good for us. It doesn't seem to be mentioned in the manual (which, contrary to the common practice, is heavily biased towards using python.withPackages), and I'm not aware of anyone actually advertising or using it. It works, but not always. Perhaps we should consider deprecating it.

I also would like to mention the case when the python interpreter binaries are not needed at all. For example, Blender is linking to the interpreter directly and needs to import and depend on python packages like numpy.

merging to fix build, if additional improvements are needed, they can be done in another PR