Add BAP to nixpkgs #24586

maurer · 2017-04-03T05:27:24Z

Motivation for this change

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Finally getting BAP into nixpkgs.
As supplementary info, see my hydra running the more extensive testsuite on it.

mention-bot · 2017-04-03T05:27:34Z

@maurer, thanks for your PR! By analyzing the history of the files in this pull request, we identified @aske, @vbgl, @sternenseemann and @FRidh to be potential reviewers.

Mic92 · 2017-04-03T07:11:27Z

pkgs/development/ocaml-modules/bap/default.nix

+  EOF
+  mkdir -p $out/lib/bap
+  make install
+  wrapProgram $out/bin/bapbuild --set OCAMLPATH `echo $OCAMLPATH` --set PATH `echo $PATH`


Excluding external OCAMLPATH and PATH is intended here? (--set instead of --prefix).

Mic92 · 2017-04-03T07:19:20Z

pkgs/development/ocaml-modules/bap/default.nix

+  export PATH=\$PATH:`echo $PATH`
+  export CAML_LD_LIBRARY_PATH=\$CAML_LD_LIBRARY_PATH:`echo $CAML_LD_LIBRARY_PATH`
+  exec ${utop}/bin/utop -ppx ppx-bap -short-paths -require "bap.top" "\$@"
+  EOF


makeWrapper would be a little more readable. There is also a --add-flags option. Also
`echo $PATH` is unnecessary: just $PATH should be enough

Mic92 · 2017-04-03T07:26:22Z

pkgs/development/ocaml-modules/bap/default.nix

+  meta = with stdenv.lib; {
+    description = "Platform for binary analysis. It is written in OCaml, but can be used from other languages.";
+    homepage = https://github.com/BinaryAnalysisPlatform/bap/;
+    license = licenses.mit;


Do you want to maintain this package?

Mic92 · 2017-04-03T07:29:56Z

pkgs/development/python-modules/bap/default.nix

+    repo = "bap-python";
+    rev = "v${version}";
+    sha256 = "0wd46ksxscgb2dci69sbndzxs6drq5cahraqq42cdk114hkrsxs3";
+  };


please add a doCheck = False; # no tests since we explicitly disable tests for python

Mic92 · 2017-04-03T07:31:21Z

pkgs/development/libraries/libbap/default.nix

+
+  preConfigure = "autoconf";
+
+  buildInputs = [ autoconf which ocaml bap findlib ctypes ];


autoconf and which belong to nativeBuildInputs

maurer · 2017-04-04T15:05:43Z

@Mic92 I believe I've addressed your concerns.

Mic92 · 2017-04-04T15:54:15Z

I added some minor changes.

maurer · 2017-04-04T16:39:58Z

@Mic92 Your changes to bap break its correctness
I can fix the indentation if you want, but those environment variables need to be set during make install or it does the wrong thing.

maurer · 2017-04-04T17:21:03Z

I reincorporated your changes (other than the postInstall, which breaks the installation) into the PR. (This version passes tests.)

Mic92 · 2017-04-04T17:48:53Z

Thanks!

vbgl · 2017-04-06T09:01:45Z

This does not build on darwin:

+ /nix/store/kvbnjil5xiscmsl0sxx27pcsy2d3q0rz-ocaml-findlib-1.7.1/bin/ocamlfind ocamlopt -linkpkg -g -I lib/bap_disasm -linkpkg -predicates used_config -predicates used_bundle -predicates used_bap_future -predicates used_plugins -predicates used_regular -predicates used_graphlib -predicates used_types -predicates used_bap_image -predicates used_disasm -predicates used_sema -predicates used_bap -predicates used_bap_byteweight -predicates ppx_driver -package zarith -package uuidm -package uri -package unix -package ppx_variants_conv -package ppx_jane -package ppx_here -package ppx_enumerate -package ocamlgraph -package findlib -package fileutils -package dynlink -package core_kernel -package cmdliner -package camlzip -package findlib.dynload -package re.posix -package curl lib/bap_bundle/bundle.cmxa lib/bap_config/config.cmxa lib/bap_future/bap_future.cmxa lib/regular/regular.cmxa lib/graphlib/graphlib.cmxa lib/bap_types/types.cmxa lib/bap_image/bap_image.cmxa lib/bap_disasm/disasm.cmxa lib/bap_plugins/plugins.cmxa lib/bap_sema/sema.cmxa lib/bap/bap.cmxa lib/bap_byteweight/bap_byteweight.cmxa src/bap_fmt_spec.cmx src/bap_source_type.cmx src/bap_options.cmx src/bap_cmdline_terms.cmx src/bap_plugin_loader.cmx src/symbols.cmx src/bap_byteweight_main.cmx -o src/bap_byteweight_main.native
ld: library not found for -lssh2

This does not build with OCaml 4.03 nor 4.04: ocamlfind: Package ‘ppx_jane’ not found.

maurer · 2017-04-06T21:58:20Z

@vgbl Those are bugs, but those aren't bugs in bap. The missing -lssh is due to a bug in the ocurl nix file, and ppx_jane has a version that works with 4.02.3 exactly (the 133.33.x series) and another version that works exclusively with 4.03+ (v0.9.0).

tl;dr

-lssh is a bug in ocurl packaging
4.03 support is dependent on someone getting ppx_jane using a supported version for 4.03+

If I find some time I can try to fix this, but it seems to make more sense to do the ppx_jane upgrade when we switch our default compiler 4.03 rather than having to do split packages again.

vbgl · 2017-04-07T08:34:07Z

Do you have a hint about what the ocurl packaging bug is?

If you just add ppx_jane to bap’s buildInputs, it will (fail to) build similarly with OCaml versions 4.02, 4.03 and 4.04. So it seems to be an issue with bap packaging.

maurer · 2017-04-09T07:26:42Z

@vgbl ocurl tries to put libcurl's linker flags into its output object files, since ocaml does all the linking at the end. I don't have a mac to test with, but I believe that pkg-config on a mac for libcurl may output -lssh, but -lssh isn't a propagated dependency. We could probably work around this by just having ocurl explicitly propagate libcurl's ssh dependency, but that seems kind of ad-hoc.

You're correct that ppx_jane should have been an input to bap, but since it's a ppx module, it actually needs to be in propagated build inputs, since anything linking against bap may use ppx_bap, which will in turn invoke ppx_jane (I'm not aware of any way to bake the path into it, see also how ppx_jane itself uses propagation for all the ppxes it bundles). I suspect that what happend for you was:

Something that declared ppx_jane as a propagated input for 4.02 build does not in a 4.03+ build
BAP didn't declare ppx_jane as a propagated input, so it went missing for 4.03+
When you added it to regular inputs, it removed it from propagated even in the 4.02 case, causing the same failure

I've tried adding it to propagated build inputs locally, and it builds+passes tests on 4.02 on my hydra, but 4.03 still fails to build, complaining about ppx_bench being missing:

+ ocamlfind ocamldep -package bap-plugin-abi -package ppx_jane -package core_kernel -package bap -modules abi.ml > abi.ml.depends
sh: /nix/store/zy3mmapa7q6xvxyqzlbwcx8jnskbj8j3-ocaml4.03.0-ppx_bench-113.33.00+4.03/lib/ocaml/4.03.0/site-lib/ppx_bench/./ppx: No such file or directory

Checking this package vs the 4.02 one,

[maurer@durandal:~/Development/nixpkgs-bap]$ ls /nix/store/zy3mmapa7q6xvxyqzlbwcx8jnskbj8j3-ocaml4.03.0-ppx_bench-113.33.00+4.03/lib/ocaml/4.03.0/site-lib/ppx_bench/
benchmark_accumulator.annot  ppx_bench.a      ppx_bench.cmt   ppx_bench.cmxs     ppx_bench_lib.cmt
benchmark_accumulator.cmt    ppx_bench.annot  ppx_bench.cmti  ppx_bench_lib.a    ppx_bench_lib.cmx
benchmark_accumulator.cmti   ppx_bench.cma    ppx_bench.cmx   ppx_bench_lib.cma  ppx_bench_lib.cmxa
META                         ppx_bench.cmi    ppx_bench.cmxa  ppx_bench_lib.cmi  ppx_bench_lib.cmxs

[maurer@durandal:~/Development/nixpkgs-bap]$ ls /nix/store/wkd5mzb45biz6pq4v946jsb3bmpw1c8j-ocaml-ppx_bench-113.33.03/lib/ocaml/4.02.3/site-lib/ppx_bench/
benchmark_accumulator.annot  META         ppx_bench.annot  ppx_bench.cmt   ppx_bench.cmxa   ppx_bench_lib.cma  ppx_bench_lib.cmx
benchmark_accumulator.cmt    ppx          ppx_bench.cma    ppx_bench.cmti  ppx_bench.cmxs   ppx_bench_lib.cmi  ppx_bench_lib.cmxa
benchmark_accumulator.cmti   ppx_bench.a  ppx_bench.cmi    ppx_bench.cmx   ppx_bench_lib.a  ppx_bench_lib.cmt  ppx_bench_lib.cmxs

[maurer@durandal:~/Development/nixpkgs-bap]$

which makes it look like the version of ppx_bench we build on 4.03 has an issue.

maurer · 2017-04-11T12:24:20Z

@vbgl See above, evidently I typo'd your handle when posting the first time.

Mic92 reviewed Apr 3, 2017

View reviewed changes

maurer force-pushed the bap branch 2 times, most recently from 64abf29 to 53f77ab Compare April 4, 2017 06:52

maurer added 3 commits April 4, 2017 13:11

bap: init at 1.2.0

7171d16

bap-python: init at 1.1.0

d022b44

libbap: init at master-2017-02-08

e010c25

maurer force-pushed the bap branch from 679b7a0 to e010c25 Compare April 4, 2017 17:18

Mic92 merged commit b04fb3f into NixOS:master Apr 4, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BAP to nixpkgs #24586

Add BAP to nixpkgs #24586

maurer commented Apr 3, 2017

mention-bot commented Apr 3, 2017

Mic92 Apr 3, 2017

Mic92 Apr 3, 2017

Mic92 Apr 3, 2017

Mic92 Apr 3, 2017

Mic92 Apr 3, 2017

maurer commented Apr 4, 2017

Mic92 commented Apr 4, 2017

maurer commented Apr 4, 2017

maurer commented Apr 4, 2017

Mic92 commented Apr 4, 2017

vbgl commented Apr 6, 2017

maurer commented Apr 6, 2017

vbgl commented Apr 7, 2017

maurer commented Apr 9, 2017

maurer commented Apr 11, 2017


		preConfigure = "autoconf";

		buildInputs = [ autoconf which ocaml bap findlib ctypes ];

Add BAP to nixpkgs #24586

Add BAP to nixpkgs #24586

Conversation

maurer commented Apr 3, 2017

Motivation for this change

Things done

mention-bot commented Apr 3, 2017

Mic92 Apr 3, 2017

Choose a reason for hiding this comment

Mic92 Apr 3, 2017

Choose a reason for hiding this comment

Mic92 Apr 3, 2017

Choose a reason for hiding this comment

Mic92 Apr 3, 2017

Choose a reason for hiding this comment

Mic92 Apr 3, 2017

Choose a reason for hiding this comment

maurer commented Apr 4, 2017

Mic92 commented Apr 4, 2017

maurer commented Apr 4, 2017

maurer commented Apr 4, 2017

Mic92 commented Apr 4, 2017

vbgl commented Apr 6, 2017

maurer commented Apr 6, 2017

vbgl commented Apr 7, 2017

maurer commented Apr 9, 2017

maurer commented Apr 11, 2017