@FRidh continuing the discussion from #63806 (comment):

Do you have any idea why pip fails to find any version of tensorflow-probability, even though it is in propagatedBuildInputs?

The package probably did not create/provide a proper dist-utils folder.

You're right, there is no distutils folder. I noticed that upstream installation instructions of tensorflow-probability want me to build it with bazel (which is annoying). But even if I do that, the problem persists.

Any idea why? How is that folder usually created?

Note it is actually the dist-info folder. This folder is created by the build-manager, so setuptools, or pip. In this case, Bazel should do that.

In the past I've encountered similar issues where upstream was using some other kind of build-system, thus not creating that folder. At the time they also provided a setuptools method for building, so I used that instead.

In your case I suppose bazel must have a way to create such a folder when install the artifact (a .whl?) with pip.

In the past I've encountered similar issues where upstream was using some other kind of build-system, thus not creating that folder. At the time they also provided a setuptools method for building, so I used that instead.

In your case I suppose bazel must have a way to create such a folder when install the artifact (a .whl?) with pip.

Yes, I can use bazel to create a wheel and then install that wheel with buildPythonPackage as their official install-from-source instruction suggest:

git clone https://github.com/tensorflow/probability.git
cd probability
bazel build --copt=-O3 --copt=-march=native :pip_pkg
PKGDIR=$(mktemp -d)
./bazel-bin/pip_pkg $PKGDIR
pip install --user --upgrade $PKGDIR/*.whl

But doing that, pip will for some reason try to replace tensorflow (which is provided as a dependency):

Requirement already satisfied: numpy>=1.13.3 in /nix/store/3ml16mnq11in68sdg8w4jipm7bqcvmh8-python3.7-numpy-1.16.4/lib/python3.7/site-packages (from tensorflow==probability)>
Installing collected packages: tensorflow
  Found existing installation: tensorflow 1.13.1
    Uninstalling tensorflow-1.13.1:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/nix/store/5vs1nfppwaigihl3ada4d68fj7ffxzdi-python3.7-tensorflow-1.13.1/bin/free>
Consider using the `--user` option or check the permissions.

If I remove tensorflow from propagatedBulidInputs it instead complains about .dist-info:

Installing collected packages: tensorflow
ERROR: Exception:
Traceback (most recent call last):
  File "/nix/store/pwqjs8nz67h4rid05vwma9z8fbsfmm16-python3.7-bootstrapped-pip-19.1.1/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main
    status = self.run(options, args)
  File "/nix/store/pwqjs8nz67h4rid05vwma9z8fbsfmm16-python3.7-bootstrapped-pip-19.1.1/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 414, in run
    use_user_site=options.use_user_site,
  File "/nix/store/pwqjs8nz67h4rid05vwma9z8fbsfmm16-python3.7-bootstrapped-pip-19.1.1/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_r>
    **kwargs
  File "/nix/store/pwqjs8nz67h4rid05vwma9z8fbsfmm16-python3.7-bootstrapped-pip-19.1.1/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 920, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/nix/store/pwqjs8nz67h4rid05vwma9z8fbsfmm16-python3.7-bootstrapped-pip-19.1.1/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 448, in move_wheel_>
    warn_script_location=warn_script_location,
  File "/nix/store/pwqjs8nz67h4rid05vwma9z8fbsfmm16-python3.7-bootstrapped-pip-19.1.1/lib/python3.7/site-packages/pip/_internal/wheel.py", line 428, in move_wheel_files
    assert info_dir, "%s .dist-info directory not found" % req
AssertionError: tensorflow==probability .dist-info directory not found

So the wheel is created, but the .dist-info dir apparently not. Also the part where it tries to replace tensorflow is curious. Any idea why?

Here's a link to their setup.py: https://github.com/tensorflow/probability/blob/master/setup.py

That's odd considering the README explicitly mentions that tensorflow is not included. Anyway, you could consider passing --no-deps to pip.

Note I'll be away for roughly a week or two.

Ah, I fixed the "no dist-info" problem. I unpacked the built wheel and noticed that the package is called tensorflow-probability (maybe picked up from pname somehow?) while the dist-info dir is called tfp-nightly because the setup.py wasn't told that it is a release.

Unfortunately to fix it I had to patch the setup.py, because as far as I can see there is no way to pass flags to pip install using buildPythonPackage. Is that right?

The attempt to install tensorflow was fixed by changing the dash in pname to an underscore. Not sure why, or what the canonical name is (I sometimes see dash, sometimes underscore with pip packages).

@GrahamcOfBorg build python3.pkgs.dm-sonnet

This is good to go from my side. @kalbasit do you want to take a look at the changes to build-bazel-package? They are a no-op for existing packages and fix the build for packages without any external dependencies (like tensorflow-probability).

To pass a flag to pip install, use installFlags. I doubt that will work here, because you will already have a wheel; the option needs to be passed when building. I think patching it is the right solution unless you can pass the flag on to Bazel.

Not sure how I could've missed installFlags, obvious in retrospect. You're right, I need to pass the flag during the wheel build and luckily that's possible. Fixed that.

Thank you for your help! I hope you'll have a nice vacation or wherever you're off to.

@kalbasit do you have any package you want to test the buildBazelPackage normalization changes on? I'm not entirely sure what the normalization is intended to do and wasn't able to find any examples.

I've mostly used it for packaging bazel-watcher, let's try it out.

@GrahamcOfBorg build bazel-watcher

ofBorg gave up, but I confirmed locally that it builds and the fixed output hash remains the same. So good to go from you @kalbasit?

nix-review shows only bazel-deps failing, but that is unrelated.

@timokau yes that's fine. I'll give it another shot, just in case.

@GrahamcOfBorg build bazel-watcher

@timokau LGTM. Please merge at your earliest convenience.

Thank you for taking a look!