python3Packages.pwntools: Allow to use any debugger #95870

Pamplemousse · 2020-08-20T18:24:51Z

Motivation for this change

pwntools allow to specify any debuggers by putting a pwntools-gdb in the PATH.
See https://github.com/Gallopsled/pwntools/blob/eb60b5d6b617d41652ab5525b7033f97e6e4c5ac/pwnlib/gdb.py#L496 .

This PR allows to declaratively specify a debugger to use with pwntools, and hooks everything as necessary; Defaults to gdb.

And, I kinda had to give gdb a pname for convenience.

Things done

Signed-off-by: Pamplemousse xav.maso@gmail.com

Pamplemousse · 2020-08-20T18:25:45Z

Questions regarding the sanity of my packaging:

It feels like the installCheckPhase does not do anything;
Is it necessary to add a pythonImportsCheck? It looks like it's done automatically (at least that's what I read in the log - Executing pythonImportsCheckPhase)...

Pamplemousse · 2020-08-20T18:40:19Z

Here is a mean to test the PR quickly in a shell (note that pwntools expects you to run inside a tmux shell to start the debugger in another window):

cat <<__EOF__ > script.py
import sys
from pwn import *

binary = sys.argv[1]
io = gdb.debug([binary])

io.interactive()
__EOF__
python script.py $(which cat)

A different debugger is used whether you started your shell with:

nix-shell -p "with import <nixpkgs> {}; pkgs.python3Packages.pwntools" - gdb;
or nix-shell -p "with import <nixpkgs> {}; pkgs.python3Packages.pwntools.override{ debugger = pkgs.pwndbg; }" - pwndbg.

pkgs/development/tools/misc/gdb/default.nix

pkgs/development/python-modules/pwntools/default.nix

kristoff3r · 2020-08-24T08:10:00Z

I'm getting errors when I run your script.py inside tmux, and outside it seems to use gdb in both cases for me. I might be doing something wrong though, I don't normally use tmux.

Error from inside tmux:

Traceback (most recent call last):
  File "script.py", line 2, in <module>
    from pwn import *
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwn/__init__.py", line 4, in <module>
    from pwn.toplevel import *
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwn/toplevel.py", line 20, in <module>
    import pwnlib
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwnlib/__init__.py", line 43, in <module>
    importlib.import_module('.%s' % module, 'pwnlib')
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwnlib/asm.py", line 58, in <module>
    from pwnlib import shellcraft
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwnlib/shellcraft/__init__.py", line 11, in <module>
    from pwnlib import constants
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwnlib/constants/__init__.py", line 61, in <module>
    from pwnlib.util import safeeval
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwnlib/util/__init__.py", line 12, in <module>
    from pwnlib.util import proc
  File "/nix/store/6n4h8xg2ldknr8909qgg5wl4kmqpm0hh-python3.8-pwntools-4.2.1/lib/python3.8/site-packages/pwnlib/util/proc.py", line 8, in <module>
    import psutil
  File "/nix/store/gafac3s76gmv5qjwilpbfn7ry5pcr30a-python3.8-psutil-5.7.2/lib/python3.8/site-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/nix/store/gafac3s76gmv5qjwilpbfn7ry5pcr30a-python3.8-psutil-5.7.2/lib/python3.8/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: cannot import name _psutil_linux

Pamplemousse · 2020-08-24T15:45:47Z

@kristoff3r, thanks for your feedback!

The way I described the tmux setup is a bit bogus...

Here is a more accurate test process:

tmux

nix-shell -p "with import <nixpkgs> {}; pkgs.python3Packages.pwntools"
python script.py $(which cat)
# This should have created a new pane with a `gdb` instance.

# Quit GDB, exit the nix-shell (but not the tmux session!)

nix-shell -p "with import <nixpkgs> {}; pkgs.python3Packages.pwntools.override{ debugger = pkgs.pwndbg; }"
python script.py $(which cat)
# This should have created a new pane with a `pwndbg` instance.

As for the error you encountered, I could not reproduce it...

kristoff3r · 2020-08-24T20:59:17Z

Thanks for the updated testing description, I've tested it and it works, so LGTM.

The error was probably because I used a non-pure shell on a Debian machine at work, I'm too used to not having python installed globally anymore.

jonringer

diff LGTM
output LGTM

however, these seem to be a series of applications. Should we do:

# all-packages.nix
  pwntools = with python3Packages; toPythonApplication pwntools;

?

jonringer · 2020-08-25T00:00:15Z

it would also prevent the propagation of python packages into an environment

Pamplemousse · 2020-08-25T22:36:35Z

@jonringer, I am not too sure about what you are asking by "should"...

It seems to "work" already (applications available in the shell); Although, I never used pwntools using the CLI.

$ nix-shell -p 'python.withPackages(ps: with ps; [ pwntools ])'
[nix-shell:~/Workspace/tools/nixpkgs]$ pwn -h
usage: pwn [-h] {asm,checksec,constgrep,cyclic,debug,disasm,disablenx,elfdiff,elfpatch,errno,hex,phd,pwnstrip,scramble,shellcraft,template,unhex,update} ...

Pwntools Command-line Interface
[...]

If you mean "should we make it cleaner", I am afraid I don't have enough knowledge of nixpkgs to answer...

~~it would also prevent the propagation of python packages into an environment~~

~~Could you explain what that means?~~
(edit) I asked some questions on IRC: https://logs.nix.samueldr.com/nixos/2020-08-25#1598395571-1598395152 , and got some help 🙂

Signed-off-by: Pamplemousse <xav.maso@gmail.com>

jonringer · 2020-08-25T23:18:01Z

the toPythonApplication allows for the commands to be used without exposing the underlying python package set. see https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#buildpythonapplication-function and https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#topythonapplication-function.

but this should allow users to do nix-shell -p pwntools --run "pwn -h"

jonringer · 2020-08-25T23:33:45Z

However, your IRC question is very different:

Could someone explain the difference between nix-shell -p python3Packages.some_package and nix-shell -p 'python.withPackages(ps: with ps; [ some_package ])' to me?

nix-shell -p python3Packages.some_package will propagate the package using $PYTHONPATH, this is useful while building a python package as you don't care to keep around the environment.

python.withPackages(ps: with ps; [ some_package ]) will do a buildEnv in which all the packages are symlinkjoin'd into a single site-packages environment. This can be thought of having an entire "virtual environment" created for the resulting python interpreter, and thus you can pass around an interpreter along with all the required packages. An example can be found

nixpkgs/pkgs/tools/misc/clipster/default.nix

Line 15 in 4dee731

pythonEnv = python3.withPackages(ps: with ps; [ pygobject3 ]);

jonringer · 2020-08-25T23:51:24Z

some code examples:

withPackages makes it look like i did pip install pandas:

$ nix-build -E 'with import <nixpkgs> { }; python37.withPackages(ps: with ps; [ pandas ])'
....
$ ls ./result/lib/python3.7/site-packages/
__pycache__                     lxml-4.5.2.dist-info      python_dateutil-2.8.1.dist-info  sqlalchemy
beautifulsoup4-4.9.1.dist-info  mock                      pytz                             SQLAlchemy-1.3.18.dist-info
bottleneck                      mock-4.0.2.dist-info      pytz-2020.1.dist-info            tables
Bottleneck-1.3.2.dist-info      numexpr                   README.txt                       tables-3.6.1.dist-info
bs4                             numexpr-2.7.1.dist-info   scipy                            webencodings
dateutil                        numpy                     scipy-1.5.2.dist-info            webencodings-0.5.1.dist-info
et_xmlfile                      numpy-1.19.1.dist-info    setuptools_scm                   xlrd
et_xmlfile-1.0.1.dist-info      openpyxl                  setuptools_scm-4.1.2.dist-info   xlrd-1.2.0.dist-info
html5lib                        openpyxl-3.0.4.dist-info  sitecustomize.py                 xlwt
html5lib-1.0.1.dist-info        pandas                    six-1.15.0.dist-info             xlwt-1.3.0.dist-info
jdcal-1.4.1.dist-info           pandas-1.1.0.dist-info    six.py
jdcal.py                        pbr                       soupsieve
lxml                            pbr-5.4.5.dist-info       soupsieve-2.0.1.dist-info

whereas, just referencing the package only makes a reference to pandas, however, the propagated-build-inputs is used to construct the PYTHONPATH when needed

$ nix-build -A python3Packages.pandas
/nix/store/lclvg47xl3jpkffpqi550glmf2cpxcny-python3.8-pandas-1.1.0
$ tree -L 4 ./result
./result
├── lib
│   └── python3.8
│       └── site-packages
│           ├── pandas
│           └── pandas-1.1.0.dist-info
└── nix-support
    └── propagated-build-inputs

Pamplemousse · 2020-08-26T15:00:58Z

@jonringer Thanks for the detailed explanation, and the links!

I had already added the line you proposed to all-packages.nix, after your comment.

jonringer

LGTM
shows usage

https://github.com/NixOS/nixpkgs/pull/95870
3 packages built:
pwntools python27Packages.pwntools python37Packages.pwntools

Pamplemousse requested review from FRidh and jonringer as code owners August 20, 2020 18:24

Pamplemousse force-pushed the pwntools branch from 2b3d3b7 to 29917c5 Compare August 20, 2020 18:24

jonringer reviewed Aug 20, 2020

View reviewed changes

pkgs/development/tools/misc/gdb/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/pwntools/default.nix Outdated Show resolved Hide resolved

Pamplemousse mentioned this pull request Aug 20, 2020

gdb: Give it a pname #95875

Merged

10 tasks

Pamplemousse changed the base branch from master to staging August 20, 2020 22:01

Pamplemousse requested review from alyssais, cdepillabout, edolstra, infinisil, matthewbauer, mmahut, nbp and ttuegel as code owners August 20, 2020 22:01

Pamplemousse changed the base branch from staging to master August 20, 2020 22:01

Pamplemousse force-pushed the pwntools branch from 29917c5 to 113c64f Compare August 21, 2020 02:56

Pamplemousse requested a review from jonringer August 21, 2020 03:07

Pamplemousse force-pushed the pwntools branch from 113c64f to e243dbb Compare August 21, 2020 03:22

ofborg bot added the 6.topic: python label Aug 21, 2020

ofborg bot requested review from bennofs and kristoff3r August 21, 2020 15:58

ofborg bot added 11.by: package-maintainer 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Aug 21, 2020

kristoff3r approved these changes Aug 24, 2020

View reviewed changes

jonringer reviewed Aug 24, 2020

View reviewed changes

python3Packages.pwntools: Allow to use any debugger

941b1e7

Signed-off-by: Pamplemousse <xav.maso@gmail.com>

Pamplemousse force-pushed the pwntools branch from e243dbb to 941b1e7 Compare August 25, 2020 22:50

ofborg bot added the 8.has: package (new) label Aug 25, 2020

ofborg bot requested a review from kristoff3r August 25, 2020 23:26

kristoff3r approved these changes Aug 26, 2020

View reviewed changes

jonringer approved these changes Aug 26, 2020

View reviewed changes

jonringer merged commit b6fc6b3 into NixOS:master Aug 26, 2020

Pamplemousse deleted the pwntools branch August 26, 2020 18:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3Packages.pwntools: Allow to use any debugger #95870

python3Packages.pwntools: Allow to use any debugger #95870

Pamplemousse commented Aug 20, 2020

Pamplemousse commented Aug 20, 2020 •

edited

Pamplemousse commented Aug 20, 2020 •

edited

kristoff3r commented Aug 24, 2020

Pamplemousse commented Aug 24, 2020

kristoff3r commented Aug 24, 2020

jonringer left a comment

jonringer commented Aug 25, 2020 •

edited

Pamplemousse commented Aug 25, 2020 •

edited

jonringer commented Aug 25, 2020

jonringer commented Aug 25, 2020

jonringer commented Aug 25, 2020

Pamplemousse commented Aug 26, 2020

jonringer left a comment

python3Packages.pwntools: Allow to use any debugger #95870

python3Packages.pwntools: Allow to use any debugger #95870

Conversation

Pamplemousse commented Aug 20, 2020

Motivation for this change

Things done

Pamplemousse commented Aug 20, 2020 • edited

Pamplemousse commented Aug 20, 2020 • edited

kristoff3r commented Aug 24, 2020

Pamplemousse commented Aug 24, 2020

kristoff3r commented Aug 24, 2020

jonringer left a comment

Choose a reason for hiding this comment

jonringer commented Aug 25, 2020 • edited

Pamplemousse commented Aug 25, 2020 • edited

jonringer commented Aug 25, 2020

jonringer commented Aug 25, 2020

jonringer commented Aug 25, 2020

Pamplemousse commented Aug 26, 2020

jonringer left a comment

Choose a reason for hiding this comment

Pamplemousse commented Aug 20, 2020 •

edited

Pamplemousse commented Aug 20, 2020 •

edited

jonringer commented Aug 25, 2020 •

edited

Pamplemousse commented Aug 25, 2020 •

edited