Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RDY] pygccxml + castxml + pybindgen #30473

Merged
merged 3 commits into from Apr 5, 2018
Merged

[RDY] pygccxml + castxml + pybindgen #30473

merged 3 commits into from Apr 5, 2018

Conversation

teto
Copy link
Member

@teto teto commented Oct 16, 2017

Motivation for this change

I use the simulator www.nsnam.org in my work. I need these libraries in order to generate the python bindings castxml + pygccxml.

Castxml linking stage fails because it can't find -ltinfo. Comments in the repository say it's not available. Not sure what to do.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@teto teto requested a review from FRidh as a code owner October 16, 2017 13:19
@@ -34,7 +34,7 @@ stdenv.mkDerivation rec {
# GHC has dtrace probes, which causes ld to try to open /usr/lib/libdtrace.dylib
# during linking
stdenv.lib.optionalString stdenv.isDarwin ''
export NIX_LDFLAGS+=" -no_dtrace_dof"
export NIX_LDFLgGS+=" -no_dtrace_dof"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh? That's probably not intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I switch between 3 different keyboards a day and sometimes I make mistakes. Will fix

lib/licenses.nix Outdated
@@ -39,6 +39,11 @@ lib.mapAttrs (n: v: v // { shortName = n; }) rec {
url = http://developer.amd.com/amd-license-agreement/;
};

apache2 = spdx {
spdxId = "AL-2.0";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://spdx.org/licenses/, that license does not exist.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't know spdxId was an official register so I took a random one. Apache 2 license is registered though https://spdx.org/licenses/Apache-2.0.html. I will fix it

@@ -1221,7 +1221,7 @@ in {
};

batinfo = buildPythonPackage rec {
version = "0.3";
version = "-1.3";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not right; the version must start with a digit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typically typo mistake too. I will mark the PR as RFC once I make sure everything works. My last commit "Now works" was a shout of joy after sphalerite helped me solve sthg I 'd spent quite some time. thanks for the review

@teto
Copy link
Member Author

teto commented Oct 17, 2017

tests fail because of an undesired warning generated by -Wunused-command-line-argument. Is that possible that the flag was added by cmake ? I looked at the cmake builder and could not find it. Might it be in clang-wrapped? tried in a nix-shell --pure with the same results. More details here
CastXML/CastXML#90 (comment)


propagatedBuildInputs = [ pkgs.castxml ];

doCheck = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you include a comment with a reason

lib/licenses.nix Outdated
@@ -39,6 +39,11 @@ lib.mapAttrs (n: v: v // { shortName = n; }) rec {
url = http://developer.amd.com/amd-license-agreement/;
};

apache2 = spdx {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the asl20

@@ -0,0 +1,70 @@
{ pkgs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not pass in pkgs

@teto teto changed the title [WIP] pygccxml + castxml [RFC] pygccxml + castxml Oct 19, 2017
@teto
Copy link
Member Author

teto commented Oct 21, 2017

I am trying to enable pygccxxml tests (previously disabled).

So first it generated

[nix-shell:~/pygccxml-v1.9.1-src]$ python -m unittests.test_all/home/teto/pygccxml-v1.9.1-src/unittests/data/core_types.hpp:9:10: fatal error: 
      'iostream' file not found
#include <iostream>
         ^~~~~~~~~~
1 error generated.
Traceback (most recent call last):
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/teto/pygccxml-v1.9.1-src/unittests/test_all.py", line 193, in <module>
    sys.exit(run_suite())
  File "/home/teto/pygccxml-v1.9.1-src/unittests/test_all.py", line 183, in run_suite
    result = unittest.TextTestRunner(verbosity=2).run(create_suite())
  File "/home/teto/pygccxml-v1.9.1-src/unittests/test_all.py", line 178, in create_suite
    main_suite.addTest(tester.create_suite())
  File "/home/teto/pygccxml-v1.9.1-src/unittests/decl_printer_tester.py", line 69, in create_suite
    suite.addTest(unittest.makeSuite(Test))
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/unittest/loader.py", line 503, in makeSuite
    testCaseClass)
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/unittest/loader.py", line 92, in loadTestsFromTestCase
    loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/unittest/suite.py", line 24, in __init__
    self.addTests(tests)
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/unittest/suite.py", line 57, in addTests
    for test in tests:
  File "/home/teto/pygccxml-v1.9.1-src/unittests/decl_printer_tester.py", line 42, in __init__
    compilation_mode=parser.COMPILATION_MODE.FILE_BY_FILE)
  File "/home/teto/pygccxml-v1.9.1-src/pygccxml/parser/project_reader.py", line 264, in read_files
    return self.__parse_file_by_file(files)
  File "/home/teto/pygccxml-v1.9.1-src/pygccxml/parser/project_reader.py", line 292, in __parse_file_by_file
    decls = reader.read_file(header)
  File "/home/teto/pygccxml-v1.9.1-src/pygccxml/parser/source_reader.py", line 356, in read_file
    return self.read_cpp_source_file(source_file)
  File "/home/teto/pygccxml-v1.9.1-src/pygccxml/parser/source_reader.py", line 375, in read_cpp_source_file
    xml_file = self.create_xml_file(ffname)
  File "/home/teto/pygccxml-v1.9.1-src/pygccxml/parser/source_reader.py", line 324, in create_xml_file
    ": %s status:%s" % (gccxml_msg, exit_status))
RuntimeError: Error occurred while running CASTXML:  status:1

so I've tried to add differents packages such as llvm.clangStdenv clang: which found some headers but not all, stdarg.h seems to be clang specific yet and though I have add clang, it's gcc includes that seem to be used ?

python -m unittests.test_allIn file included from /home/teto/pygccxml-v1.9.1-src/unittests/data/core_types.hpp:9:
In file included from /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/iostream:39:
In file included from /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/ostream:38:
In file included from /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/ios:38:
In file included from /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/iosfwd:40:
In file included from /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/bits/postypes.h:40:
In file included from /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/cwchar:44:
/nix/store/qxb0x9fcs60zv719yy0zm95gyd8mmgks-glibc-2.25-49-dev/include/wchar.h:40:11: fatal error: 
      'stdarg.h' file not found
# include <stdarg.h>
          ^~~~~~~~~~
1 error generated.
Traceback (most recent call last):
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/2lb6nn8ivk1alhckv43n7734lqwbw7h9-python3-3.6.2/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/teto/pygccxml-v1.9.1-src/unittests/test_all.py", line 193, in <module>
    sys.exit(run_suite())
  File "/home/teto/pygccxml-v1.9.1-src/unittests/test_all.py", line 183, in run_suite

should I set CXXFLAGS myself or can a package do it for me ?

@teto teto force-pushed the pygccxml branch 3 times, most recently from bc26ff3 to 17ad38c Compare October 24, 2017 08:01
@teto teto mentioned this pull request Nov 7, 2017
8 tasks
@Mic92
Copy link
Member

Mic92 commented Nov 7, 2017

Getting the include path for standard c++ headers is a bit tricky. The following expression might help:

$ nix-instantiate --eval --expr 'with (import <nixpkgs>) {}; clang.default_cxx_stdlib_compile'
"-isystem $(echo -n /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/include/c++/*) -isystem $(echo -n /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/include/c++/*)/$(/nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/bin/gcc -dumpmachine)"

The linking problem with -ltinfo can be solved by a symlink to ncurses (example: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/science/programming/fdr/default.nix#L30)

@teto
Copy link
Member Author

teto commented Nov 8, 2017

I changed the checkPhase to

   checkPhase = ''
    export CFLAGS="${clang.default_cxx_stdlib_compile}"
    echo $CFLAGS
    python -m unittests.test_all
  '';

but the code would still fail to find stdarg.h, I searched the echoed CFLAGS for stdarg.h

$ fd stdarg.h /nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0                                       
/nix/store/rww78vdn2rkayrnqsjl8ib5iq2vfm3sw-gcc-6.4.0/include/c++/6.4.0/tr1/stdarg.h

and it is there but in a subfolder. The homepage https://github.com/gccxml/pygccxml doesn't list clang compatibility (only python). Maybe I need to add a flag in CFLAG so that stdarg.h appears in the upper folder. Like -std=c++11 ?

@teto
Copy link
Member Author

teto commented Dec 6, 2017

I tried enabling pygccxml tests but it proved kinda hard, environment variables are not useful so one needs to generate a file and the include_paths is checked so that -isystem $(echo -n /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/include/c++/*) -isystem $(echo -n /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/include/c++/*)/$(/nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/bin/gcc -dumpmachine) is not ok.

Meanwhile it would be cool if it could be merged without the tests as I need it for my work.

PS: Here is what I had in my attempt to enable tests (for my future self or anyone interested in taking over)

  xml_generator = writeText "xml_generator.cfg" ''
    [xml_generator]
    # Specify which xml generator you want to use "castxml" or "gccxml"
    # "castxml is the default"
    xml_generator=castxml
    # Path to castxml or gccxml executable file
    xml_generator_path=${castxml}/bin/castxml
    # Set the path to the compiler (for example "/usr/bin/gcc") for CastXML
    compiler_path=${llvmPackages.clang-unwrapped}/bin/clang++
    # Gccxml working directory - optional, could be set to your source code directory
    working_directory=
    # Additional include directories, as list of paths ["path1/file1.h", "path2/file2.h", ...]
    include_paths=
    # You can explicitly set what compiler it should emulate (for GCCXML)
    # Valid options are: g++, msvc6, msvc7, msvc71, msvc8, cl.
    compiler=
    # Keep xml files after errors (useful for debugging)
    keep_xml=1

    '';

  checkPhase = ''
    # export CFLAGS="-I${llvmPackages.clang-unwrapped.out}/lib/clang/4.0.1/include"
 # - if [[ $CPPSTD == "-std=c++14" ]]; then
 #      mv unittests/configs/travis_std14.cfg unittests/xml_generator.cfg;
# fi
    # echo $CFLAGS
    cp ${xml_generator} unittests/xml_generator.cfg
    python3 -m unittests.test_all
  '';
debug  checkInputs = [which sphinx llvmPackages.libcxxClang pycodestyle llvmPackages.clang-unwrapped];

4 tests are ignored because the regex don't match with custom clang
output see:
CastXML/CastXML#90
@teto
Copy link
Member Author

teto commented Apr 5, 2018

@GrahamcOfBorg build pythonPackages.pybindgen pythonPackages.pygccxml castxml

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: pythonPackages.pybindgen, pythonPackages.pygccxml, castxml

Partial log (click to expand)

running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s


OK
/nix/store/p1v8y9g1yrbab2m22h48m05wf42rhyf0-python2.7-PyBindGen-0.18.0
/nix/store/9gsqlkirlp26462bckajjy03xg4fzx06-python2.7-pygccxml-1.9.1
/nix/store/67sl6kydgifzjv37sh7kzzwv5g71cli7-CastXML-20180403

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: pythonPackages.pybindgen, pythonPackages.pygccxml, castxml

Partial log (click to expand)

writing manifest file 'PyBindGen.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/wrnfb9766fhc0k2bayk2bmz3n92s4lns-python2.7-PyBindGen-0.18.0
/nix/store/hqqn0rjy5la9m9939zw8dgghxbna4jv6-python2.7-pygccxml-1.9.1
/nix/store/sik3875plq6pb8vd4gkja3w4j5ig1975-CastXML-20180403

@teto
Copy link
Member Author

teto commented Apr 5, 2018

@GrahamcOfBorg build python3Packages.pybindgen python3Packages.pygccxml castxml

@teto teto changed the title [RFC] pygccxml + castxml [RFC] pygccxml + castxml + pybindgen Apr 5, 2018
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python3Packages.pybindgen, python3Packages.pygccxml, castxml

Partial log (click to expand)

writing manifest file 'PyBindGen.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/vsqdazl4s4za7ijixmdfw3r0yv51sfvk-python3.6-PyBindGen-0.18.0
/nix/store/nf4zxdvs615i0gvj9r9br6gchgvxd02p-python3.6-pygccxml-1.9.1
/nix/store/67sl6kydgifzjv37sh7kzzwv5g71cli7-CastXML-20180403

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python3Packages.pybindgen, python3Packages.pygccxml, castxml

Partial log (click to expand)

writing manifest file 'PyBindGen.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK
/nix/store/qs0nh9vczw5l2iivhs29ng86rj17d3xi-python3.6-PyBindGen-0.18.0
/nix/store/hhcxwhznf58k1cfkygq0plp4lsv6bvl3-python3.6-pygccxml-1.9.1
/nix/store/sik3875plq6pb8vd4gkja3w4j5ig1975-CastXML-20180403

teto added 2 commits April 5, 2018 19:15
Python library to play with castxml output, aka your program's clang-produced AST.
Helps generating python bindings to C. Required to generate
www.nsnam.org bindings.
@teto teto changed the title [RFC] pygccxml + castxml + pybindgen [RDY] pygccxml + castxml + pybindgen Apr 5, 2018
@teto
Copy link
Member Author

teto commented Apr 5, 2018

I added pybindgen so as to support python bindings in #31346 (they released ns3.28 and fixed some problems in the build so I can finally update that PR).
The maintainer of pygccxml clearly stated that the tests were tailored for its CI & when updating castxml, it went from 4 to 96 errors mostly because it checks the generated gcc command lines and nixos adds its own path -isystem ... that confuses the regexes. So I disabled the tests in both case.
I marked the PR as ready.

@Mic92 Mic92 merged commit 129fbf9 into NixOS:master Apr 5, 2018
@Mic92
Copy link
Member

Mic92 commented Apr 5, 2018

Thanks!

@teto teto deleted the pygccxml branch April 5, 2018 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants