I also did this here: https://github.com/NixOS/nixpkgs/pull/40426/files :)

Your pull request has some changes, I should incorporate like the output.

Mhm. I think we cannot importe bcc into python packages since it depends on the kernel version.

Do we really need a dedicated python output? Bcc already depends on python anyway.

ah nice that you put up a package. I am not that knowledgeable with bcc, I wanted to use it to trace and collect statistics on some kernel functions rather than adding printk's. Your python.pkgs.buildPythonApplication might be a better approach then. Feel free to close this

on second thought with the buildPythonApplication, will I be able to import bcc and write my own apps ?

on second thought with the buildPythonApplication, will I be able to import bcc and write my own apps ?

No.

@FRidh are you sure? the PYTHONPATH was set correctly in a nix-shell:

$ nix-shell -p linuxPackages.bcc
bash: /run/user/1000/env-vars: cannot overwrite existing file
[nix-shell:~/git/nixpkgs]$ echo $PYTHONPATH
/nix/store/d90chi5qw76bbncy4b1b0qb2m065c7yh-bcc-0.5.0/lib/python3.6/site-packages:...
[nix-shell:~/git/nixpkgs]$ python
Python 3.6.5 (default, Mar 28 2018, 10:24:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bcc

Also the following snippet works fine:

with import <nixpkgs> {};
python3.pkgs.buildPythonApplication {
  name = "bcc-app";
  format = "other";
  unpackPhase = ":";
  installPhase = ''
    mkdir -p $out/bin/
    cat > $out/bin/script <<EOF
    #!/usr/bin/env python
    import bcc
    print("hello")
    EOF
    chmod +x $out/bin/script
  '';
  propagatedBuildInputs = [ linuxPackages.bcc ];
}

$ nix-build
$ result/bin/script
hello

So I would say, yes, you can use my version to build your own applications on top of bcc.

Technically it works with the current implementation but it is not supposed to (#11423). How would one use this with withPackages, which is the preferred solution? We should not be importing from top-level packages.

Note that with withPackages/buildEnv I intentionally broke support for it. Until 3 years ago we had a lot of Python modules in the top-level package set, and that caused a lot of trouble.

I suppose toPythonModule can be used for withPackages. We cannot easily import bcc into pythonPackages. It strongly depend on the kernel version/header that is used as it compiles at runtime bpf code that is executed in the kernel.

We cannot easily import bcc into pythonPackages. It strongly depend on the kernel version/header that is used as it compiles at runtime bpf code that is executed in the kernel.

How is that any different for buildPythonPackage than for buildPythonApplication?

I thought your concern was about importing from top-level.

It is, I thought you were referring to having to use buildPythonPackage.

But, the actual issue is, as you pointed out, that it is made for a certain kernel version. Is it a problem that users of the Python library would have to use override in order to pass in a different kernel? There is no way around that, really, anyway.

I would have liked to use nix-shell -p python.withPackages as a shebang but I suppose it won't work.
I've cherry-picked #40426 and ran sudo nix-shell -p python '(linuxPackagesFor my_lenovo_kernel).bcc my_lenovo_kernel.dev (my_lenovo_kernel being a custom kernel). Is that the correct way to get bcc as a package ?
Then I successfully started the bcc hello world.

nix-shell:/home/teto/testbed]# python hello_world.py
      python3.6m-1781  [003] ....  5249.621283: 0: Hello, World!
              sh-18805 [000] ....  5249.627813: 0: Hello, World!
              sh-18805 [000] ....  5249.627883: 0: Hello, World!
      python3.6m-1781  [003] ....  5254.626520: 0: Hello, World!
              sh-18808 [000] ....  5254.632796: 0: Hello, World!
              sh-18808 [000] ....  5254.632863: 0: Hello, World!