Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vowpalwabbit: fix build against boost-python. #45987

Merged
merged 3 commits into from Sep 4, 2018
Merged

vowpalwabbit: fix build against boost-python. #45987

merged 3 commits into from Sep 4, 2018

Conversation

danieldk
Copy link
Contributor

@danieldk danieldk commented Sep 3, 2018 •
edited

Patch setup.py to look for libboost_python{Major}{Minor}. This fixes the build in unstable and 18.09.

Motivation for this change

Going through the list of #45960 I found that the Vowpal Wabbit Python module does not compile. This fixes compilation with Python 2.7 and 3.6.

Note: for some reason, boost is added to the build environment without boost-python, unless I explicitly add python.pkgs.boost as a dependency.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@danieldk
vowpalwabbit: fix build against boost-python.
875bb6b 
Patch setup.py to look for libboost_python{Major}{Minor}.
@danieldk danieldk requested a review from FRidh as a code owner September 3, 2018 13:51
@vcunat
Copy link
Member

vcunat commented Sep 3, 2018

@GrahamcOfBorg build python2Packages.vowpalwabbit python3Packages.vowpalwabbit

@vcunat
Copy link
Member

vcunat commented Sep 3, 2018

/cc maintainer @teh.

@FRidh
Copy link
Member

FRidh commented Sep 3, 2018

Note: for some reason, boost is added to the build environment without boost-python, unless I explicitly add python.pkgs.boost as a dependency.

That's indeed what should be used.

@FRidh
Copy link
Member

FRidh commented Sep 3, 2018

@GrahamcOfBorg build python2Packages.vowpalwabbit python3Packages.vowpalwabbit

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

    self._build_extensions_serial()
  File "/nix/store/dlswl7xjvh3bsfydygqzi2n0f82p5d9p-python3-3.6.6/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 114, in build_extension
    env['PYTHON_LIBS'] = '-l {}'.format(find_boost())
  File "setup.py", line 39, in find_boost
    raise Exception('Could not find boost python library')
Exception: Could not find boost python library
builder for '/nix/store/vfvjr3lfs9c10gp5yddlzhfmx4fr7gjw-python3.6-vowpalwabbit-8.5.0.drv' failed with exit code 1
error: build of '/nix/store/fq9xcrp3x7lwlg30j73cfq5yalwcj88i-python2.7-vowpalwabbit-8.5.0.drv', '/nix/store/vfvjr3lfs9c10gp5yddlzhfmx4fr7gjw-python3.6-vowpalwabbit-8.5.0.drv' failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

    self._build_extensions_serial()
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 115, in build_extension
    subprocess.check_call(['make', 'python'], cwd=path.join(here, 'src'), env=env)
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', 'python']' returned non-zero exit status 2.
builder for '/nix/store/yyryl97m3xcpzk5a524rf0jk4qsgvbn1-python3.6-vowpalwabbit-8.5.0.drv' failed with exit code 1
error: build of '/nix/store/9p5k6dq76kvpdzgcjbzr35lmw99wrk94-python2.7-vowpalwabbit-8.5.0.drv', '/nix/store/yyryl97m3xcpzk5a524rf0jk4qsgvbn1-python3.6-vowpalwabbit-8.5.0.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0
shrinking /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0/lib/python3.6/site-packages/pylibvw.cpython-36m-x86_64-linux-gnu.so
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0/lib
patching script interpreter paths in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0
checking for references to /build in /nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0...
running install tests
running check
/nix/store/n6qm2ygsvb25nr7rmsq6rpkmhs6cvxfp-python2.7-vowpalwabbit-8.5.0
/nix/store/4jqjjk5cdcs6psha5wrpb41xn0fqxryf-python3.6-vowpalwabbit-8.5.0

@danieldk
vowpal-wabbit: fix Darwin build.
fe7e89b 
- Follow the same logic for finding boost-python as for Linux.
- Set DYLD_LIBRARY_PATH so that ctypes find_library can find
  boost-python.
- Do not use '-mt' suffix when linking against boost-serialization
  and boost-program_options on Darwin.
@danieldk
Copy link
Contributor Author

danieldk commented Sep 3, 2018

The added commit should also fix the build on Darwin (tested on macOS 10.14 beta).

@xeji
Copy link
Contributor

xeji commented Sep 3, 2018

@GrahamcOfBorg build python2Packages.vowpalwabbit python3Packages.vowpalwabbit

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

    self._build_extensions_serial()
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "setup.py", line 113, in build_extension
    subprocess.check_call(['make', 'python'], cwd=path.join(here, 'src'), env=env)
  File "/nix/store/sb88r9crdhinhkm6vpxwzf5zmij86p8v-python3-3.6.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', 'python']' returned non-zero exit status 2.
builder for '/nix/store/68w132wx8hfhpiq0b3rfqrfz5z02wik9-python3.6-vowpalwabbit-8.5.0.drv' failed with exit code 1
error: build of '/nix/store/68w132wx8hfhpiq0b3rfqrfz5z02wik9-python3.6-vowpalwabbit-8.5.0.drv', '/nix/store/827xwn1q8ygmvy0pa2gya2x13kvqr0lp-python2.7-vowpalwabbit-8.5.0.drv' failed

@xeji
Copy link
Contributor

xeji commented Sep 3, 2018

This never built on aarch64 according to hydra logs, so would you please mark it as broken on aarch64 to get rid of one more failure?

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0
shrinking /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0/lib/python3.6/site-packages/pylibvw.cpython-36m-x86_64-linux-gnu.so
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0/lib
patching script interpreter paths in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0
checking for references to /build in /nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0...
running install tests
running check
/nix/store/jy6d5va2gbm7dhj42lvyx09pjf3h1a3y-python2.7-vowpalwabbit-8.5.0
/nix/store/lh0ri1z0wv5b88rm4qxzrhwagk8sh1j2-python3.6-vowpalwabbit-8.5.0

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: python2Packages.vowpalwabbit, python3Packages.vowpalwabbit

Partial log (click to expand)

Successfully installed vowpalwabbit-8.5.0
~
post-installation fixup
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/jp3cdh73crchd25d2m26d3g75yhmp0bz-python3.6-vowpalwabbit-8.5.0/lib
patching script interpreter paths in /nix/store/jp3cdh73crchd25d2m26d3g75yhmp0bz-python3.6-vowpalwabbit-8.5.0
running install tests
running check
/nix/store/94i1pvdzkbl5f8jyrik01m01n3s8jh0d-python2.7-vowpalwabbit-8.5.0
/nix/store/jp3cdh73crchd25d2m26d3g75yhmp0bz-python3.6-vowpalwabbit-8.5.0

@danieldk
Copy link
Contributor Author

danieldk commented Sep 4, 2018

Ok, marked it broken on aarch64.

@xeji xeji merged commit d22e22e into NixOS:master Sep 4, 2018
@xeji
Copy link
Contributor

xeji commented Sep 4, 2018

backported in 46f4a93..b447979

@danieldk danieldk deleted the fix-vowpalwabbit branch September 5, 2018 14:25
@danieldk danieldk restored the fix-vowpalwabbit branch September 5, 2018 14:25
@danieldk danieldk deleted the fix-vowpalwabbit branch September 5, 2018 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@danieldk @vcunat @FRidh @GrahamcOfBorg @xeji