Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

truffleHog: init at 2.0.91 #40391

Merged
merged 4 commits into from May 14, 2018
Merged

truffleHog: init at 2.0.91 #40391

merged 4 commits into from May 14, 2018

Conversation

bhipple
Copy link
Contributor

@bhipple bhipple commented May 12, 2018

This commit adds the python3 application truffleHog, which is a stand-alone tool
that scans a git repo for unencrypted passwords.

This depends on a newer GitPython, which depends on a new major version of
gitdb, which depends on a new major version of smmap2, so I've packaged those
as well.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@bhipple bhipple requested a review from FRidh as a code owner May 12, 2018 01:25
bhipple
bhipple commented May 12, 2018
View reviewed changes
pkgs/tools/security/truffleHog/default.nix Outdated
# Relax overly restricted version constraint
postPatch = ''
sed -i "s/GitPython == 2.1.1/GitPython >= 2.1.1/" setup.py
'';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC @FRidh is there a better way to loosen overly conservative semver bounds? I did build all of these packages, and verified that truffleHog is working fine with the newer GitPython.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly, there isn't. But using

sed -i "s/GitPython ==/GitPython >=/" setup.py

could reduce the pain upon updating.
Also, consider using substituteInPlace.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: truffleHog

Partial log (click to expand)

Successfully installed truffleHog-2.0.91
/build/truffleHog-2.0.91
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91/lib  /nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91/bin
patching script interpreter paths in /nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91
checking for references to /build in /nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91...
wrapping `/nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91/bin/trufflehog'...
/nix/store/0nmcba3cl2vgd51gqrvwy65rfl51krgv-truffleHog-2.0.91

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: truffleHog

Partial log (click to expand)

Successfully installed truffleHog-2.0.91
/build/truffleHog-2.0.91
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91
strip is /nix/store/gp7fylxwn18b7pl2c18ks89hsiaxyfvf-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91/lib  /nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91/bin
patching script interpreter paths in /nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91
checking for references to /build in /nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91...
wrapping `/nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91/bin/trufflehog'...
/nix/store/idgw64adsh2mj9i7p410pp13z5jpkh28-truffleHog-2.0.91

@dotlambda
Copy link
Member

dotlambda commented May 12, 2018 •
edited

Would you mind moving GitPython, gitdb2 and smmap2 out of python-packages.nix and creating seperate commits for those?

dotlambda
dotlambda reviewed May 12, 2018
View reviewed changes
pkgs/tools/security/truffleHog/default.nix Outdated
# Relax overly restricted version constraint
postPatch = ''
sed -i "s/GitPython == 2.1.1/GitPython >= 2.1.1/" setup.py
'';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly, there isn't. But using

sed -i "s/GitPython ==/GitPython >=/" setup.py

could reduce the pain upon updating.
Also, consider using substituteInPlace.

pkgs/tools/security/truffleHog/default.nix Outdated
buildInputs = [ ];
propagatedBuildInputs = [ GitPython truffleHogRegexes ];

doCheck = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment why tests are disabled.

pkgs/tools/security/truffleHog/default.nix Outdated
sed -i "s/GitPython == 2.1.1/GitPython >= 2.1.1/" setup.py
'';

buildInputs = [ ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop this line

pkgs/top-level/all-packages.nix Outdated
@@ -21110,6 +21110,8 @@ with pkgs;

tup = callPackage ../development/tools/build-managers/tup { };

truffleHog = python3.pkgs.callPackage ../tools/security/truffleHog { };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please pass python3 or python3Packages as an argument instead. This will allow overriding of specific dependencies' versions if necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this?
truffleHog = pythonPackages.callPackage ../tools/security/truffleHog { };

pkgs/top-level/python-packages.nix Outdated
};

buildInputs = with self; [ mock nose ];
propagatedBuildInputs = with self; [ gitdb ];
buildInputs = with self; [ mock nose ddt ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checkInputs

pkgs/top-level/python-packages.nix Outdated
sha256 = "1hvn28p3zvxa98sbi9lrqvv2ps4q284j4jq9a619zw0m7yv0sly7";
};

buildInputs = with self; [ nosexcover ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checkInputs

FRidh
FRidh requested changes May 12, 2018
View reviewed changes
pkgs/top-level/python-packages.nix Outdated
@@ -2843,20 +2843,39 @@ in {
homepage = https://github.com/gitpython-developers/gitdb;
license = licenses.bsd3;
};
};

gitdb2 = buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are moving Python expressions out of pkgs/top-level/python-packages.nix into pkgs/development/python-modules/<module>/default.nix.

Please move the expression there, and call it from pkgs/top-level/python-packages.nix using callPackage ../development/python-modules/<package> { };.

pkgs/top-level/python-packages.nix Outdated
maintainers = with maintainers; [ ];
homepage = https://github.com/gitpython-developers/gitdb;
license = licenses.bsd3;
};
};

GitPython = buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also move this while you're at it.

pkgs/top-level/python-packages.nix Outdated
@@ -14291,6 +14310,24 @@ in {
};
};

smmap2 = buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are moving Python expressions out of pkgs/top-level/python-packages.nix into pkgs/development/python-modules/<module>/default.nix.

Please move the expression there, and call it from pkgs/top-level/python-packages.nix using callPackage ../development/python-modules/<package> { };.

@bhipple bhipple force-pushed the truffleHog-gitpython-gitdb2-smmap2 branch from 59ef98f to 1db90ab Compare May 12, 2018 15:01
@bhipple
Copy link
Contributor Author

bhipple commented May 12, 2018 •
edited

Thx guys; the newer python module structure looks a lot cleaner and nicer! I've refactored this into four commits: the first one moves smmap2, the second gitdb2, the third GitPython, and the last one truffleHog.

smmap2 and gitdb2 come from the same repo as smmap and gitdb, but they're different packages on pypi and have different import names. It looks like GitPython is the only one that was using these, at least within nixpkgs; I know what generally prefer to keep as few versions as possible (ideally just 1), but for majors do we usually drop them or leave them around in case someone is using them? At least within NixPkgs I think it'd be OK to delete the smmap and gitdb packages.

@bhipple bhipple force-pushed the truffleHog-gitpython-gitdb2-smmap2 branch 2 times, most recently from 604839e to fd614c9 Compare May 12, 2018 15:13
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: truffleHog

Partial log (click to expand)

Installing collected packages: truffleHog
Successfully installed truffleHog-2.0.91
/build/truffleHog-2.0.91
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/biz06ll6li8alshxhmy05442qw0xnfp4-truffleHog-2.0.91
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/biz06ll6li8alshxhmy05442qw0xnfp4-truffleHog-2.0.91/lib  /nix/store/biz06ll6li8alshxhmy05442qw0xnfp4-truffleHog-2.0.91/bin
patching script interpreter paths in /nix/store/biz06ll6li8alshxhmy05442qw0xnfp4-truffleHog-2.0.91
checking for references to /build in /nix/store/biz06ll6li8alshxhmy05442qw0xnfp4-truffleHog-2.0.91...
wrapping `/nix/store/biz06ll6li8alshxhmy05442qw0xnfp4-truffleHog-2.0.91/bin/trufflehog'...

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: truffleHog

Partial log (click to expand)

Successfully installed truffleHog-2.0.91
/build/truffleHog-2.0.91
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91
strip is /nix/store/gp7fylxwn18b7pl2c18ks89hsiaxyfvf-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91/lib  /nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91/bin
patching script interpreter paths in /nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91
checking for references to /build in /nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91...
wrapping `/nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91/bin/trufflehog'...
/nix/store/3jd06ccjdiwr03b9qi2wg5p2g9kfwf42-truffleHog-2.0.91

FRidh
FRidh requested changes May 13, 2018
View reviewed changes
pkgs/development/python-modules/smmap/default.nix Outdated
};
}

# smmap2 = buildPythonPackage rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

old, needs to be removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah thanks, forgot to remove that; I had initially thought about putting them both in one file, since they come from the same git repo and share the same meta info, but decided against it to keep the consistent directory name -> python import name.

Removed.

pkgs/top-level/all-packages.nix Outdated
@@ -21110,6 +21110,8 @@ with pkgs;

tup = callPackage ../development/tools/build-managers/tup { };

truffleHog = pythonPackages.callPackage ../tools/security/truffleHog { };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you keep the attribute entirely lowercase, and the same for the folder of the package.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this should simply be callPackage, and the trufflehog expressions then has a parameter python instead of the individual Python packages and functions that are needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lowercased and swapped to callPackage; thanks! Should I reword the commit message to trufflehog: init at 2.0.91, or leave it consistent with the pypi name?

@bhipple
smmap2: init at 2.0.3
68573a6 
This commit adds smmap2, which is a separate pypi package from the existing
smmap 0.9.0, though they both come from the same repo. It also moves smmap into
the new python modules location.
@bhipple
gitdb2: init at 2.0.3
318ce72 
Also move gitdb and gitdb2 to the new python module structure.
@bhipple
GitPython: 2.0.8 -> 2.1.9
5efce7c 
Also moves GitPython into the newer python module folder structure.
@bhipple bhipple force-pushed the truffleHog-gitpython-gitdb2-smmap2 branch from fd614c9 to 3a91dc1 Compare May 13, 2018 15:35
@bhipple
truffleHog: init at 2.0.91
3fc1694 
This commit adds the python3 application truffleHog, which is a stand-alone tool
that scans a git repo for unencrypted passwords.

This depends on a newer GitPython, which depends on a new major version of
gitdb, which depends on a new major version of smmap, so I've packaged those
as well in the preceding commits.
@bhipple bhipple force-pushed the truffleHog-gitpython-gitdb2-smmap2 branch from 3a91dc1 to 3fc1694 Compare May 13, 2018 15:41
@matthewbauer matthewbauer merged commit e5274eb into NixOS:master May 14, 2018
@bhipple bhipple deleted the truffleHog-gitpython-gitdb2-smmap2 branch May 19, 2018 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotlambda dotlambda dotlambda left review comments

@FRidh FRidh FRidh requested changes

Assignees
No one assigned
Labels
6.topic: python 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@bhipple @GrahamcOfBorg @dotlambda @FRidh @matthewbauer