Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages.caffe: fix caffe python bindings #63324

Merged
merged 3 commits into from Jul 20, 2019
Merged

Conversation

avitex
Copy link
Member

@avitex avitex commented Jun 17, 2019

Motivation for this change

Fixes #63191

cc @jb55

I have not tested with python 2 fully, only confirmed the python 3 bindings are now working (though I will, when I get time).
If you'd like to add me as a maintainer, I'm happy to review future problems

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@avitex avitex changed the title Fix caffe python bindings pythonPackages.caffe: fix caffe python bindings Jun 18, 2019
Copy link
Contributor

@jb55 jb55 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this!

outputs = [ "bin" "out"];
propagatedBuildInputs = lib.optionals pythonSupport (
# requirements.txt
let pp = python.pkgs; in ([
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not with python.pkgs [ numpy scipy ... ]?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did try that initially, but protobuf seemed to clash with the above import (didn't install).
Perhaps I messed up something else, but this was my workaround

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please cherry-pick matthiasbeyer@879db97

With this patch, caffe still builds for me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build works regardless of the clash, simply start up python and import caffe.

Python 3.7.3 (default, Mar 25 2019, 20:59:09)
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/30n7msvdrqhj4ab9151rp402j50vdsck-caffe-1.0/lib/python3.7/site-packages/caffe/__init__.py", line 4, in <module>
    from .proto.caffe_pb2 import TRAIN, TEST
  File "/nix/store/30n7msvdrqhj4ab9151rp402j50vdsck-caffe-1.0/lib/python3.7/site-packages/caffe/proto/caffe_pb2.py", line 7, in <module>
    from google.protobuf.internal import enum_type_wrapper
ModuleNotFoundError: No module named 'google'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you even import this, the package seems to provide only binaries, no libraries... or am I completely missing something here?

Copy link
Member Author

@avitex avitex Jun 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to specify python3 as an extra package, python3Packages.xxx internally requires it.
I'm on nixos 19.03. Try adding the pure arg --pure to nix-shell. Your patch is not working for me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ nix-shell -p python3Packages.caffe --pure --run "python"

And then importing caffe works for me, I'm on centos7 on my testing machine. Fun things... maybe someone can explain why this works for me...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're running centos 7, do you have python bindings protobuf elsewhere in the env? rpm -q protobuf-python perhaps? Maybe try importing google and print(google.__file__). Aside from that, I have no idea

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, none of that. Also, --pure would prevent leaking stuff into the environment, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this flag is specified, the environment is almost entirely cleared before the interactive shell is started, so you get an environment that more closely corresponds to the “real” Nix build. A few variables, in particular HOME, USER and DISPLAY, are retained. Note that ~/.bashrc and (depending on your Bash installation) /etc/bashrc are still sourced, so any variables set there will affect the interactive shell.

AKA, if PYTHONPATH is leaking in somehow, that might explain why.
I'm off to sleep now friend. G'night

@FRidh
Copy link
Member

FRidh commented Jul 20, 2019

tested locally with

$ nix run -f . "(import ./. {}).python3.withPackages(ps: with ps; [ caffe])" --command python3 -c "import caffe"

Python 2 builds, there is just an unrelated collision.

@FRidh FRidh merged commit 44324bd into NixOS:master Jul 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Caffe python bindings seem to be broken
5 participants