pythonPackages.binwalk: add dependencies #94268

Pamplemousse · 2020-07-30T23:39:02Z

Motivation for this change

Allow binwalk to extract CramFS content.
Here is the error I get with the current version:

WARNING: Extractor.execute failed to run external extractor 'cramfsck -x 'cramfs-root' '%e'': [Errno 2] No such file or directory: 'cramfsck', 'cramfsck -x 'cramfs-root'
 '%e'' might not be installed correctly                                             

WARNING: Extractor.execute failed to run external extractor 'cramfsswap '%e' '%e.swap' && cramfsck -x 'cramfs-root' '%e.swap'': [Errno 2] No such file or directory: 'cra
mfsswap', 'cramfsswap '%e' '%e.swap' && cramfsck -x 'cramfs-root' '%e.swap'' might not be installed correctly
128           0x80            CramFS filesystem, little endian, size: 9912320, version 2, sorted_dirs, CRC 0xCA750C65, edition 0, 5736 blocks, 607 files

This comment explicitly states that cramfsck and cramfsswap are required to work.

Things done

pkgs/os-specific/linux/cramfsprogs/default.nix

jonringer · 2020-07-31T06:49:36Z

This seems like you should package cramfsck and cramfsswap as applications then pass them to the python wrapper so they are available.

I'm not sure if the current PR would solve the issue. IIRC, python packages which are passed through propagatedBuildInputs don't propagate their commands, just the packages.

Mic92 · 2020-07-31T09:24:06Z

pkgs/development/python-modules/binwalk/default.nix

+  propagatedBuildInputs = [ zlib xz ncompress gzip bzip2 gnutar p7zip cabextract cramfsswap cramfsprogs lzma pycrypto ]
  ++ stdenv.lib.optional visualizationSupport pyqtgraph;


I have not checked the other tools. I suspect they also should be in PATH rather than
propagatedBuildInputs.

Suggested change

propagatedBuildInputs = [ zlib xz ncompress gzip bzip2 gnutar p7zip cabextract cramfsswap cramfsprogs lzma pycrypto ]

++ stdenv.lib.optional visualizationSupport pyqtgraph;

propagatedBuildInputs = [ zlib xz ncompress gzip bzip2 gnutar p7zip cabextract lzma pycrypto ]

++ stdenv.lib.optional visualizationSupport pyqtgraph;

makeWrapperArgs = ["--prefix" "PATH" ":" (stdenv.lib.makeBinPath [ cramfsswap cramfsprogs ])];

I don't understand what this addition is supposed to do:
When I nix-shell --pure -p 'python3.withPackages(ps: with ps; [binwalk])', the $PATH is not changed whether I this line (makeWrapperArgs...) is present in the expression or not.

@jonringer, you made a similar proposition below: What is that supposed to do?

I am not too sure I am testing that correctly 😕: I don't see any difference when I am adding the aforementioned line:

# cramfsswap is still not available via the binwalk package $ nix-shell --pure -p 'python3.withPackages(ps: with ps; [binwalk])' --run "cramfsswap" /tmp/nix-shell-13495-0/rc: line 1: cramfsswap: command not found # while it works when imported itself $ nix-shell --pure -p cramfsswap --run "cramfsswap" Usage: cramfsswap <in> <out>

Poke @Mic92 , @jonringer .

@Pamplemousse you don't want to change the PATH of your nix-shell but the PATH of the binwalk executable. If you execute binwalk from the nix store directly without nix-shell, than cramfs won't be in PATH. Note that nix-shell also packs propagated build inputs into your PATH, however those are not proper runtime dependencies but build dependencies instead.

Does this makes sense?

@Mic92 , thanks for the explanation. So, if I understand correctly:
Using the makeWrapperArgs ... allows to solve the error (aforementioned in the first comment) both when invoking binwalk from a nix-shell, but also from the executable in the nix store.

As for the second part of your remark, I am not quite sure I get the full implications of what you say:

So, adding cramfsswap cramfsprogs to the propagatedBuildInputs do not make them runtime dependencies... Is it only incidental that without makeWrapperArgs, and using nix-shell (because they are added to its PATH) the error was solved when I tested?

I kinda hoped those binaries to be available in the shell when importing binwalk, is that an over-expectation? (i.e. it is "normal" for import of python modules not to expose in the nix-shell the binaries they rely on)

Pamplemousse · 2020-07-31T14:23:29Z

@jonringer Interesting...

It did "solve my issue" (i.e., I can extract the content of a CramFs image using binwalk).

$ nix-shell -p 'python3.withPackages(ps: with ps; [binwalk])' --run "binwalk -e firmwares/FW_BU-2015_1.03.18_2014-06-05"

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
128           0x80            CramFS filesystem, little endian, size: 9912320, version 2, sorted_dirs, CRC 0xCA750C65, edition 0, 5736 blocks, 607 files

However, cramfsswap and cramfsck are not available:

$ nix-shell -p 'python3.withPackages(ps: with ps; [binwalk])' --run "cramfsswap"
/tmp/nix-shell-20130-0/rc: line 1: cramfsswap: command not found
$ nix-shell -p 'python3.withPackages(ps: with ps; [binwalk])' --run "cramfsck" 
/tmp/nix-shell-20177-0/rc: line 1: cramfsck: command not found

I wonder how binwalk ended up working at all 🤔

Pamplemousse · 2020-07-31T16:36:46Z

I added a commit to update the expression of cramfsswap for the binary to be usable:

# before
$ nix-shell --pure -p 'cramfsswap' --run 'cramfsswap -h'/nixpkgs
/tmp/nix-shell-10374-0/rc: line 1: cramfsswap: command not found

# after
$ nix-shell --pure -p 'cramfsswap' --run 'cramfsswap -h'
Usage: cramfsswap <in> <out>

jonringer

I could be wrong about the propagatedBuildInputs and PATH, but I still think it would be preferable to do something like:

  makeWrapperArgs = [ "--prefix PATH : ${makeBinPath [ cramfsswap cramfsprogs ]} ';

As I have seen large dev closures when propagating native packages.

Fortunately, this doesnt seem to be the case here:

$ nix path-info -Sh ./result
/nix/store/6h3cq7dk6g3v0zq235sa8plgafifqv5k-python3.8-binwalk-2.2.0	 160.0M
[nix-shell:/home/jon/.cache/nixpkgs-review/pr-94268]$ nix path-info -Sh ./results/python37Packages.binwalk
/nix/store/xxcdvxp77lzwxp8a4gwkap0qb3rdzrkz-python3.7-binwalk-2.2.0	 157.7M

pkgs/os-specific/linux/cramfsprogs/include-sysmacros.patch

pkgs/os-specific/linux/cramfsprogs/default.nix

jonringer · 2020-07-31T16:39:23Z

I think you pushed some commits by accident

Signed-off-by: Pamplemousse <xav.maso@gmail.com>

jonringer

LGTM
shows usage

https://github.com/NixOS/nixpkgs/pull/94268
8 packages built:
cramfsprogs cramfsswap python27Packages.binwalk python27Packages.binwalk-full python37Packages.binwalk python37Packages.binwalk-full python38Packages.binwalk python38Packages.binwalk-full

Pamplemousse requested review from FRidh and jonringer as code owners July 30, 2020 23:39

ofborg bot added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 30, 2020

Mic92 reviewed Jul 31, 2020

View reviewed changes

pkgs/os-specific/linux/cramfsprogs/default.nix Outdated Show resolved Hide resolved

Mic92 reviewed Jul 31, 2020

View reviewed changes

pkgs/os-specific/linux/cramfsprogs/default.nix Outdated Show resolved Hide resolved

Mic92 reviewed Jul 31, 2020

View reviewed changes

Pamplemousse force-pushed the binwalk branch from efb22c4 to f1070f0 Compare July 31, 2020 16:36

jonringer reviewed Jul 31, 2020

View reviewed changes

pkgs/os-specific/linux/cramfsprogs/include-sysmacros.patch Show resolved Hide resolved

pkgs/os-specific/linux/cramfsprogs/default.nix Outdated Show resolved Hide resolved

Pamplemousse force-pushed the binwalk branch from f1070f0 to db3b277 Compare July 31, 2020 17:06

Pamplemousse added 3 commits July 31, 2020 10:26

cramfsprogs: init at 1.1

8cb4bd7

Signed-off-by: Pamplemousse <xav.maso@gmail.com>

cramfsswap: Make binary available

6d2d5b3

Signed-off-by: Pamplemousse <xav.maso@gmail.com>

pythonPackages.binwalk: add dependencies

f611a00

Signed-off-by: Pamplemousse <xav.maso@gmail.com>

Pamplemousse force-pushed the binwalk branch from db3b277 to f611a00 Compare July 31, 2020 17:26

jonringer approved these changes Aug 7, 2020

View reviewed changes

jonringer merged commit 35c702c into NixOS:master Aug 7, 2020

Pamplemousse deleted the binwalk branch August 20, 2020 15:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.binwalk: add dependencies #94268

pythonPackages.binwalk: add dependencies #94268

Pamplemousse commented Jul 30, 2020

jonringer commented Jul 31, 2020

Mic92 Jul 31, 2020

Pamplemousse Jul 31, 2020

Pamplemousse Jul 31, 2020

Pamplemousse Aug 6, 2020

Mic92 Aug 7, 2020

Mic92 Aug 7, 2020

Pamplemousse Aug 7, 2020

Pamplemousse commented Jul 31, 2020 •

edited

Pamplemousse commented Jul 31, 2020

jonringer left a comment •

edited

jonringer commented Jul 31, 2020

jonringer left a comment

		propagatedBuildInputs = [ zlib xz ncompress gzip bzip2 gnutar p7zip cabextract cramfsswap cramfsprogs lzma pycrypto ]
		++ stdenv.lib.optional visualizationSupport pyqtgraph;

pythonPackages.binwalk: add dependencies #94268

pythonPackages.binwalk: add dependencies #94268

Conversation

Pamplemousse commented Jul 30, 2020

Motivation for this change

Things done

jonringer commented Jul 31, 2020

Mic92 Jul 31, 2020

Choose a reason for hiding this comment

Pamplemousse Jul 31, 2020

Choose a reason for hiding this comment

Pamplemousse Jul 31, 2020

Choose a reason for hiding this comment

Pamplemousse Aug 6, 2020

Choose a reason for hiding this comment

Mic92 Aug 7, 2020

Choose a reason for hiding this comment

Mic92 Aug 7, 2020

Choose a reason for hiding this comment

Pamplemousse Aug 7, 2020

Choose a reason for hiding this comment

Pamplemousse commented Jul 31, 2020 • edited

Pamplemousse commented Jul 31, 2020

jonringer left a comment • edited

Choose a reason for hiding this comment

jonringer commented Jul 31, 2020

jonringer left a comment

Choose a reason for hiding this comment

Pamplemousse commented Jul 31, 2020 •

edited

jonringer left a comment •

edited