[RFC] pythonPackages.libnl: init+bump at 3.4.0 #40470

teto · 2018-05-14T03:11:23Z

Netlink is the userspace <-> kernelspace coomunication protocol.

Motivation for this change

I need it
Note: I don't know if it's because netlink examples are outdated or because of my packaging but
import netlink.genl.capi, one would need to do import netlink.genl._capi. Maybe I should rename _capi.so to capi.so ?

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Mic92 · 2018-05-14T11:23:15Z

It looks like we have a name collision here. Is something else importing libnl in pythonPackages?
There should not be this many rebuilds.

Mic92 · 2018-05-14T11:24:25Z

pkgs/development/python-modules/libnl/default.nix

+
+  nativeBuildInputs = libnl.nativeBuildInputs ++ [  swig file ];
+
+  propagatedBuildInputs = [ libnl ];


This should not be required if libnl is used to build an internal binding.

Mic92 · 2018-05-14T11:25:48Z

pkgs/development/python-modules/libnl/default.nix

+  '';
+
+  postConfigure=''
+    cd python


Maybe we should have a python output in libnl instead? What python versions does libnl support at the moment?

Mic92 · 2018-05-14T11:27:26Z

@FRidh might have a better idea how optional python bindings in libraries is handled better.

teto · 2018-05-15T02:07:56Z

It looks like we have a name collision here. Is something else importing libnl in pythonPackages?
There should not be this many rebuilds.

The rebuild is triggered only if the derivation .drv file changes right ? could my addition of the version field in the derivation have triggered this ? many things depend on netlink (not sure though if they depend specifically from libnl though).

Mic92 · 2018-05-15T08:54:38Z

All packages in python that do not take libnl from pkgs. can potentially inherit the wrong package.

FRidh · 2018-05-15T08:56:34Z

Typically we build packages with bindings, and put those in a separate output. We can then wrap that output in python-packages.nix. There are some examples there.

Mic92 · 2018-05-15T09:00:41Z

Could you give an example? I have not found any example so far.

teto · 2018-05-15T09:06:51Z

fontforge/gpgpme ? but those 2 have specifc targets for bindings. Anyway I am now modifying the package. Should I add python to outputs = [ "bin" "dev" "out" "man" "python" ]; inconditionally and/or remove the folder when !withPython ? 2018-05-15 18:00 GMT+09:00 Jörg Thalheim <notifications@github.com>:

…

Could you give an example? I have not found any example so far. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#40470 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2FOkMQbA_Pd_6DTRQw1Ijf4fabqS2oks5typk_gaJpZM4T9GgU> .

FRidh · 2018-05-15T09:12:46Z

Related issue regarding bindings #20516.

Look for toPythonModule.

Some examples:

boost
opencv
librepo
libplist
libxml2
libxslt
rivet

teto · 2018-05-15T10:55:04Z

The examples you've shown seem to all have the binding generation part of their buildsystem. That's not the case for libnl so I will still need some kind of python packaging (to run python tests)
I ended up moving the python folder to a py output but it just contains the source, not a python package.
Also there seems to be tests to run too. I managed to

{ buildPythonPackage, pkgconfig, src, isPy3k, libnl, swig, file }:
buildPythonPackage {
  pname = "libnl";
  inherit (libnl) version  patches meta;

  src=libnl.py;

  disabled = isPy3k;

  buildInputs = [ pkgconfig swig libnl libnl.dev ];
}

what I don't get though is why the python package won't compile even though I provided the libnl.dev output aong with pkgconfig; the correct include path (sthg like /nix/store/d29dwbldfg8qsgcsgp4ic14av9iw5rlw-libnl-3.4.0-dev/include/libnl3) won't appear in gcc command line:


building 'netlink/_capi' extension
swigging ./netlink/capi.i to ./netlink/capi_wrap.c
swig -python -O -nodefaultctor -o ./netlink/capi_wrap.c ./netlink/capi.i
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/netlink
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../include -I../include -I/nix/store/l9j2jsc9flrbmpf799nw9wdq59gpwms8-python-2.7.14/include/python2.7 -c ./netlink/capi_wrap.c -o build/temp.linux-x86_64-2.7/./netlink/capi_wrap.o
./netlink/capi_wrap.c:3070:10: fatal error: netlink/netlink.h: No such file or directory
 #include <netlink/netlink.h>
          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
note: keeping build directory '/tmp/nix-build-python2.7-libnl-3.4.0.drv-0'

teto · 2018-05-17T12:09:13Z

I think this addresses the concern: I run the setup.py install with --prefix and that works. I renamed the python bindings to libnl-python to prevent name clashes
nix-shell -p 'python.withPackages(ps: with ps; [ libnl-python ])' --run "python -c 'import netlink'" should work

GrahamcOfBorg · 2018-05-17T12:19:18Z

Success on x86_64-linux (full log)

Attempted: libnl

Partial log (click to expand)

strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/hvvj1rprkfl4qkkkwsw9gz8p9f48l025-libnl-3.4.0/lib
patching script interpreter paths in /nix/store/hvvj1rprkfl4qkkkwsw9gz8p9f48l025-libnl-3.4.0
checking for references to /build in /nix/store/hvvj1rprkfl4qkkkwsw9gz8p9f48l025-libnl-3.4.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/z5lyxx3x8z9pfamxqba3n4s8a7y1f147-libnl-3.4.0-man
gzipping man pages under /nix/store/z5lyxx3x8z9pfamxqba3n4s8a7y1f147-libnl-3.4.0-man/share/man/
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/z5lyxx3x8z9pfamxqba3n4s8a7y1f147-libnl-3.4.0-man
checking for references to /build in /nix/store/z5lyxx3x8z9pfamxqba3n4s8a7y1f147-libnl-3.4.0-man...
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip

GrahamcOfBorg · 2018-05-17T12:27:49Z

Success on aarch64-linux (full log)

Attempted: libnl

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/3gj71lgsh14wskxm0207a0i96fz2pnnw-libnl-3.4.0/lib
patching script interpreter paths in /nix/store/3gj71lgsh14wskxm0207a0i96fz2pnnw-libnl-3.4.0
checking for references to /build in /nix/store/3gj71lgsh14wskxm0207a0i96fz2pnnw-libnl-3.4.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/ga635qzbypxlf4xi7knfajpvjrf35y9w-libnl-3.4.0-man
gzipping man pages under /nix/store/ga635qzbypxlf4xi7knfajpvjrf35y9w-libnl-3.4.0-man/share/man/
strip is /nix/store/jk6j4lh9v5mvjdbdc35sj0zffhhf6s56-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/ga635qzbypxlf4xi7knfajpvjrf35y9w-libnl-3.4.0-man
checking for references to /build in /nix/store/ga635qzbypxlf4xi7knfajpvjrf35y9w-libnl-3.4.0-man...
strip is /nix/store/jk6j4lh9v5mvjdbdc35sj0zffhhf6s56-binutils-2.30/bin/strip
/nix/store/xlqhggsvar0md4rp6m2nfjpgm0b2innr-libnl-3.4.0-bin

teto · 2018-05-17T12:43:37Z

pkgs/os-specific/linux/libnl/default.nix

+
+  postBuild = lib.optionalString (pythonSupport) ''
+      cd python
+      ${python}/bin/python setup.py install --prefix=../pythonlib


maybe I could just use --prefix=$py straightaway ?

Try it.

It's a pity building a wheel does not work.

${python.interpreter} instead of ${python}/bin/python

I tried it and it generates the folder so that I don't need the postFixup phase anymore \o/

FRidh · 2018-05-19T04:50:26Z

pkgs/os-specific/linux/libnl/default.nix

+  '';
+
+  postFixup = lib.optionalString pythonSupport ''
+    mv "pythonlib/" "$py"


use moveToOutput

I had tried several times but I could never get moveToOutput to do anything. I wonder if I don't get the usag right.

anyway this hook is not needed anymore

GrahamcOfBorg · 2018-05-19T05:29:41Z

Success on x86_64-linux (full log)

Attempted: libnl

Partial log (click to expand)

patching script interpreter paths in /nix/store/1zx706hivamg2p79430n29rjj1dm3kz9-libnl-3.4.0-man
checking for references to /build in /nix/store/1zx706hivamg2p79430n29rjj1dm3kz9-libnl-3.4.0-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py
shrinking /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py/lib/python2.7/site-packages/netlink/route/_capi.so
shrinking /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py/lib/python2.7/site-packages/netlink/genl/_capi.so
shrinking /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py/lib/python2.7/site-packages/netlink/_capi.so
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py/lib
patching script interpreter paths in /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py
checking for references to /build in /nix/store/7dssqjwm8kyw9zg8z9prpr2v6g42ivzk-libnl-3.4.0-py...

GrahamcOfBorg · 2018-05-19T05:33:20Z

Success on aarch64-linux (full log)

Attempted: libnl

Partial log (click to expand)

checking for references to /build in /nix/store/g9ykdbfa8gaacc80w41qybq94yspzl08-libnl-3.4.0-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py
shrinking /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py/lib/python2.7/site-packages/netlink/genl/_capi.so
shrinking /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py/lib/python2.7/site-packages/netlink/route/_capi.so
shrinking /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py/lib/python2.7/site-packages/netlink/_capi.so
strip is /nix/store/jk6j4lh9v5mvjdbdc35sj0zffhhf6s56-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py/lib
patching script interpreter paths in /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py
checking for references to /build in /nix/store/08p2afxgfpmrpzm5g181ymnys4zvjcj8-libnl-3.4.0-py...
/nix/store/bq6pfvzdwcd6rqvz9z0z9n2k8iaycdfi-libnl-3.4.0-bin

Netlink is the userspace <-> kernelspace communication protocol.

GrahamcOfBorg · 2018-05-21T10:14:08Z

Success on x86_64-linux (full log)

Attempted: libnl

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/6g3i8xz29yl74y421wkv36ci87n329sm-libnl-3.4.0/lib
patching script interpreter paths in /nix/store/6g3i8xz29yl74y421wkv36ci87n329sm-libnl-3.4.0
checking for references to /build in /nix/store/6g3i8xz29yl74y421wkv36ci87n329sm-libnl-3.4.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/rm0hcspvs18qj8qf95q8i7xp09cnr3cp-libnl-3.4.0-man
gzipping man pages under /nix/store/rm0hcspvs18qj8qf95q8i7xp09cnr3cp-libnl-3.4.0-man/share/man/
strip is /nix/store/mgyvr3kil9qv0mywikpi0dbli061bpg4-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/rm0hcspvs18qj8qf95q8i7xp09cnr3cp-libnl-3.4.0-man
checking for references to /build in /nix/store/rm0hcspvs18qj8qf95q8i7xp09cnr3cp-libnl-3.4.0-man...
strip is /nix/store/mgyvr3kil9qv0mywikpi0dbli061bpg4-binutils-2.30/bin/strip
/nix/store/6r6f9840sw1a0yh6r1c7f8bsn70zwsk5-libnl-3.4.0-bin

Mic92 · 2018-05-21T10:33:20Z

@FRidh package looks good to me. What do you say?

GrahamcOfBorg · 2018-05-21T11:53:44Z

Success on aarch64-linux (full log)

Attempted: libnl

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/xa4d4dirgbrsj84lzdiwlv8rwzl6zkvw-libnl-3.4.0/lib
patching script interpreter paths in /nix/store/xa4d4dirgbrsj84lzdiwlv8rwzl6zkvw-libnl-3.4.0
checking for references to /build in /nix/store/xa4d4dirgbrsj84lzdiwlv8rwzl6zkvw-libnl-3.4.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/452crwdd7pcxw40vafhbk7gflnvzl872-libnl-3.4.0-man
gzipping man pages under /nix/store/452crwdd7pcxw40vafhbk7gflnvzl872-libnl-3.4.0-man/share/man/
strip is /nix/store/8yfik687kfccisxnad42j19lfb7ij9b4-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/452crwdd7pcxw40vafhbk7gflnvzl872-libnl-3.4.0-man
checking for references to /build in /nix/store/452crwdd7pcxw40vafhbk7gflnvzl872-libnl-3.4.0-man...
strip is /nix/store/8yfik687kfccisxnad42j19lfb7ij9b4-binutils-2.30/bin/strip
/nix/store/306ji4as6nj66szw3zkqwq69f7z8gavb-libnl-3.4.0-bin

teto · 2018-05-21T12:11:57Z

thanks for the guidance.

teto requested a review from FRidh as a code owner May 14, 2018 03:11

GrahamcOfBorg added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 501+ labels May 14, 2018

Mic92 reviewed May 14, 2018

View reviewed changes

teto force-pushed the libnl_python branch from 044be99 to a8c177c Compare May 17, 2018 12:03

GrahamcOfBorg added the 2.status: merge conflict label May 17, 2018

teto force-pushed the libnl_python branch from a8c177c to 8dbfb67 Compare May 17, 2018 12:07

GrahamcOfBorg removed the 2.status: merge conflict label May 17, 2018

teto changed the title ~~[RFC] pythonPackages.libnl: init at 3.3.0~~ [RFC] pythonPackages.libnl: init+bump at 3.4.0 May 17, 2018

teto commented May 17, 2018

View reviewed changes

FRidh reviewed May 19, 2018

View reviewed changes

teto force-pushed the libnl_python branch from 8dbfb67 to ce81096 Compare May 19, 2018 05:16

teto changed the base branch from master to staging May 21, 2018 09:50

teto added 2 commits May 21, 2018 18:51

libnl: 3.3.0 -> 3.4.0

d9dadce

pythonPackages.libnl-python: init at 3.4.0

0d17a57

Netlink is the userspace <-> kernelspace communication protocol.

teto force-pushed the libnl_python branch from ce81096 to 0d17a57 Compare May 21, 2018 09:52

teto mentioned this pull request May 21, 2018

Guidance for packaging packages providing Python bindings #40835

Open

Mic92 merged commit 99c1d03 into NixOS:staging May 21, 2018

teto deleted the libnl_python branch May 21, 2018 12:11

teto mentioned this pull request Oct 10, 2018

python: update documentation #48132

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] pythonPackages.libnl: init+bump at 3.4.0 #40470

[RFC] pythonPackages.libnl: init+bump at 3.4.0 #40470

teto commented May 14, 2018 •

edited

Mic92 commented May 14, 2018

Mic92 May 14, 2018

Mic92 May 14, 2018

Mic92 commented May 14, 2018

teto commented May 15, 2018

Mic92 commented May 15, 2018

FRidh commented May 15, 2018

Mic92 commented May 15, 2018

teto commented May 15, 2018 via email

FRidh commented May 15, 2018

teto commented May 15, 2018

teto commented May 17, 2018

GrahamcOfBorg commented May 17, 2018

GrahamcOfBorg commented May 17, 2018

teto May 17, 2018

FRidh May 19, 2018

teto May 19, 2018

FRidh May 19, 2018

teto May 19, 2018

teto May 19, 2018 •

edited

GrahamcOfBorg commented May 19, 2018

GrahamcOfBorg commented May 19, 2018

GrahamcOfBorg commented May 21, 2018

Mic92 commented May 21, 2018

GrahamcOfBorg commented May 21, 2018

teto commented May 21, 2018


		nativeBuildInputs = libnl.nativeBuildInputs ++ [ swig file ];

		propagatedBuildInputs = [ libnl ];

[RFC] pythonPackages.libnl: init+bump at 3.4.0 #40470

[RFC] pythonPackages.libnl: init+bump at 3.4.0 #40470

Conversation

teto commented May 14, 2018 • edited

Motivation for this change

Things done

Mic92 commented May 14, 2018

Mic92 May 14, 2018

Choose a reason for hiding this comment

Mic92 May 14, 2018

Choose a reason for hiding this comment

Mic92 commented May 14, 2018

teto commented May 15, 2018

Mic92 commented May 15, 2018

FRidh commented May 15, 2018

Mic92 commented May 15, 2018

teto commented May 15, 2018 via email

FRidh commented May 15, 2018

teto commented May 15, 2018

teto commented May 17, 2018

GrahamcOfBorg commented May 17, 2018

GrahamcOfBorg commented May 17, 2018

teto May 17, 2018

Choose a reason for hiding this comment

FRidh May 19, 2018

Choose a reason for hiding this comment

teto May 19, 2018

Choose a reason for hiding this comment

FRidh May 19, 2018

Choose a reason for hiding this comment

teto May 19, 2018

Choose a reason for hiding this comment

teto May 19, 2018 • edited

Choose a reason for hiding this comment

GrahamcOfBorg commented May 19, 2018

GrahamcOfBorg commented May 19, 2018

GrahamcOfBorg commented May 21, 2018

Mic92 commented May 21, 2018

GrahamcOfBorg commented May 21, 2018

teto commented May 21, 2018

teto commented May 14, 2018 •

edited

teto May 19, 2018 •

edited