Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages.dodgy: init at 0.2.1 #77255

Merged
merged 1 commit into from Jan 9, 2020
Merged

pythonPackages.dodgy: init at 0.2.1 #77255

merged 1 commit into from Jan 9, 2020

Conversation

kamadorueda
Copy link
Member

@kamadorueda kamadorueda commented Jan 7, 2020

Motivation for this change

dodgy is a nice tool to check for secrets in source code

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

nix-review rev HEAD plus bynary test
kamado:~/Documents/nixpkgs$ nix-review rev HEAD
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nix-review/0
From https://github.com/NixOS/nixpkgs
   0b7b588de24..7d7d41f7b5e  master     -> refs/nix-review/0
$ git worktree add /home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/nixpkgs 7d7d41f7b5e0e8339384ecd9a1d6837b0df3902a
Preparing worktree (detached HEAD 7d7d41f7b5e)
HEAD is now at 7d7d41f7b5e Merge pull request #74912 from edef1c/pounce
$ nix-env -f /home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit f4e8dbce45692e3862c106311630532763bb51fc
Updating 7d7d41f7b5e..f4e8dbce456
Fast-forward
 pkgs/development/python-modules/dodgy/default.nix | 33 +++++++++++++++++++++++++++++++++
 pkgs/top-level/all-packages.nix                   |  2 ++
 pkgs/top-level/python-packages.nix                |  2 ++
 3 files changed, 37 insertions(+)
 create mode 100644 pkgs/development/python-modules/dodgy/default.nix
$ nix-env -f /home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/nixpkgs -qaP --xml --out-path --show-trace --meta
$ nix build --no-link --keep-going --max-jobs 4 --option build-use-sandbox true -f /home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/build.nix
[1 built, 0.0 MiB DL]
2 package were built:
dodgy python38Packages.dodgy

$ nix-shell /home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/shell.nix
these paths will be fetched (1.51 MiB download, 8.27 MiB unpacked):
  /nix/store/6n34bphaagi53sv2zfk3x56z5b5vbccd-bash-interactive-4.4-p23-man
  /nix/store/c3pjljc2ha0jspsban2lfalqkzv2ncn5-readline-7.0p5
  /nix/store/g1fpiz69s4l8yk4gjsly5hwbr3nm64i5-bash-interactive-4.4-p23-info
  /nix/store/ixhlw5k9ziazafxkn74wvg36kbv5vl4f-bash-interactive-4.4-p23-doc
  /nix/store/jw8pwmn1g55pymkmshcwghyv2z13k5n3-bash-interactive-4.4-p23-dev
  /nix/store/nd2irl4gl044zq9dsrb3c3chsr13cb2k-bash-interactive-4.4-p23
copying path '/nix/store/ixhlw5k9ziazafxkn74wvg36kbv5vl4f-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/g1fpiz69s4l8yk4gjsly5hwbr3nm64i5-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/6n34bphaagi53sv2zfk3x56z5b5vbccd-bash-interactive-4.4-p23-man' from 'https://cache.nixos.org'...
copying path '/nix/store/c3pjljc2ha0jspsban2lfalqkzv2ncn5-readline-7.0p5' from 'https://cache.nixos.org'...
copying path '/nix/store/nd2irl4gl044zq9dsrb3c3chsr13cb2k-bash-interactive-4.4-p23' from 'https://cache.nixos.org'...
copying path '/nix/store/jw8pwmn1g55pymkmshcwghyv2z13k5n3-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
innovation

[nix-shell:~/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc]$ dodgy 
Unable to read '/home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/nixpkgs/pkgs/tools/archivers/zip/natspec-gentoo.patch.bz2': 'utf-8' codec can't decode byte 0xd4 in position 10: invalid continuation byte
Unable to read '/home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/nixpkgs/pkgs/tools/X11/xbindkeys-config/xbindkeys-config-patch1.patch': 'utf-8' codec can't decode byte 0xe9 in position 312: invalid continuation byte
Unable to read '/home/kamado/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc/nixpkgs/pkgs/games/rili/moderinze_cpp.patch': 'utf-8' codec can't decode byte 0xe9 in position 3689: invalid continuation byte
{
  "warnings": [
    {
      "path": "nixpkgs/pkgs/servers/mail/mailman/settings.py",
      "line": 43,
      "code": "secret",
      "message": "Possible hardcoded secret key"
    }
  ]
}

[nix-shell:~/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc]$ dodgy --help
usage: dodgy [-h] [--ignore-paths IGNORE_PATH [IGNORE_PATH ...]] [--zero-exit]

A very basic tool to run against your codebase to search for "dodgy" looking
values. It is a series of simple regular expressions designed to detect things
such as accidental SCM diff checkins, or passwords/secret keys hardcoded into
files.

optional arguments:
  -h, --help            show this help message and exit
  --ignore-paths IGNORE_PATH [IGNORE_PATH ...], -i IGNORE_PATH [IGNORE_PATH ...]
                        Paths to ignore
  --zero-exit, -0       Dodgy will exit with a code of 1 if problems are
                        found. This flag ensures that it always returns with 0
                        unless an exception is raised.

[nix-shell:~/.cache/nix-review/rev-f4e8dbce45692e3862c106311630532763bb51fc]$ 

@kamadorueda
Copy link
Member Author

kamadorueda commented Jan 7, 2020

lol actually what dodgy found is real, a django secret key:

nixpkgs/pkgs/servers/mail/mailman/settings.py

image

should I change that key for a randomly-generated-at-runtime one?

pkgs/development/python-modules/dodgy/default.nix Outdated Show resolved Hide resolved
Comment on lines 20 to 21
# Tests are not passing:
# $ git clone git@github.com:landscapeio/dodgy.git
# $ tox
doCheck = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tox just abstracts over python interpreter versions, taken from tox.ini:

  checkInputs = [ nose mock ];
  checkPhase = ''
    nosetests -s
  '';

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that's true, however, for real the tests are not passing, (it's not nix related, it's related to the actual dodgy code), see:

test are broken from the very source code:

kamado:~/Documents/dodgy$ tox
GLOB sdist-make: /home/kamado/Documents/dodgy/setup.py
py34 create: /home/kamado/Documents/dodgy/.tox/py34
ERROR: InterpreterNotFound: python3.4
py35 create: /home/kamado/Documents/dodgy/.tox/py35
ERROR: InterpreterNotFound: python3.5
py36 create: /home/kamado/Documents/dodgy/.tox/py36
ERROR: InterpreterNotFound: python3.6
py37 inst-nodeps: /home/kamado/Documents/dodgy/.tox/.tmp/package/1/dodgy-0.2.2.zip
py37 installed: dodgy==0.2.2,mock==3.0.5,nose==1.3.7,six==1.13.0
py37 run-test-pre: PYTHONHASHSEED='1738575116'
py37 run-test: commands[0] | nosetests -s
......{
  "warnings": [
    "should-be-warning"
  ]
}
{
  "warnings": []
}
EEE
======================================================================
ERROR: Failure: ModuleNotFoundError (No module named 'dodgy.configuration')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kamado/Documents/dodgy/tests/test_configuration.py", line 3, in <module>
    from dodgy.configuration import get_options
ModuleNotFoundError: No module named 'dodgy.configuration'

======================================================================
ERROR: test_return_1_for_warnings (tests.test_run.TestRun)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/site-packages/mock/mock.py", line 1330, in patched
    return func(*args, **keywargs)
  File "/home/kamado/Documents/dodgy/tests/test_run.py", line 17, in test_return_1_for_warnings
    self.assertEqual(run(), 1)
  File "/home/kamado/Documents/dodgy/dodgy/run.py", line 73, in run
    sys.exit(1 if warnings else 0)
SystemExit: 1

======================================================================
ERROR: test_return_zero_for_success (tests.test_run.TestRun)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kamado/Documents/dodgy/.tox/py37/lib/python3.7/site-packages/mock/mock.py", line 1330, in patched
    return func(*args, **keywargs)
  File "/home/kamado/Documents/dodgy/tests/test_run.py", line 12, in test_return_zero_for_success
    self.assertEqual(run(), 0)
  File "/home/kamado/Documents/dodgy/dodgy/run.py", line 73, in run
    sys.exit(1 if warnings else 0)
SystemExit: 0

----------------------------------------------------------------------
Ran 9 tests in 0.016s

FAILED (errors=3)
ERROR: InvocationError for command /home/kamado/Documents/dodgy/.tox/py37/bin/nosetests -s (exited with code 1)
py38 inst-nodeps: /home/kamado/Documents/dodgy/.tox/.tmp/package/1/dodgy-0.2.2.zip
py38 installed: dodgy==0.2.2,mock==3.0.5,nose==1.3.7,six==1.13.0
py38 run-test-pre: PYTHONHASHSEED='1738575116'
py38 run-test: commands[0] | nosetests -s
......{
  "warnings": [
    "should-be-warning"
  ]
}
{
  "warnings": []
}
EEE
======================================================================
ERROR: Failure: ModuleNotFoundError (No module named 'dodgy.configuration')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kamado/Documents/dodgy/tests/test_configuration.py", line 3, in <module>
    from dodgy.configuration import get_options
ModuleNotFoundError: No module named 'dodgy.configuration'

======================================================================
ERROR: test_return_1_for_warnings (tests.test_run.TestRun)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/site-packages/mock/mock.py", line 1330, in patched
    return func(*args, **keywargs)
  File "/home/kamado/Documents/dodgy/tests/test_run.py", line 17, in test_return_1_for_warnings
    self.assertEqual(run(), 1)
  File "/home/kamado/Documents/dodgy/dodgy/run.py", line 73, in run
    sys.exit(1 if warnings else 0)
SystemExit: 1

======================================================================
ERROR: test_return_zero_for_success (tests.test_run.TestRun)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kamado/Documents/dodgy/.tox/py38/lib/python3.8/site-packages/mock/mock.py", line 1330, in patched
    return func(*args, **keywargs)
  File "/home/kamado/Documents/dodgy/tests/test_run.py", line 12, in test_return_zero_for_success
    self.assertEqual(run(), 0)
  File "/home/kamado/Documents/dodgy/dodgy/run.py", line 73, in run
    sys.exit(1 if warnings else 0)
SystemExit: 0

----------------------------------------------------------------------
Ran 9 tests in 0.016s

FAILED (errors=3)
ERROR: InvocationError for command /home/kamado/Documents/dodgy/.tox/py38/bin/nosetests -s (exited with code 1)
_______________________________________________________ summary _______________________________________________________
ERROR:  py34: InterpreterNotFound: python3.4
ERROR:  py35: InterpreterNotFound: python3.5
ERROR:  py36: InterpreterNotFound: python3.6
ERROR:   py37: commands failed
ERROR:   py38: commands failed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

their tests are failing, however dodgy works fine, I suppose tests/source-code got un-syncrhonized at some point

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  checkInputs = [ nose mock ];
  checkPhase = ''
    nosetests tests/test_checks.py
  '';

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like test_run.py and test_configuration.py require some external setup logic, just scope it to what runs locally

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice idea!

just force-pushed the branch with the changes

@jonringer
Copy link
Contributor

cc @peti you may want to look at this:

SECRET_KEY = '$!-7^wl#wiifjbh)5@f7ji%x!vp7s1vzbvwt26hxv$idixq0u0'

@kamadorueda
Copy link
Member Author

all suggestions have been applied:

kamado:~/Documents/nixpkgs$ nix-review rev HEAD
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nix-review/0
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 7 (delta 3), reused 5 (delta 3), pack-reused 0
Unpacking objects: 100% (7/7), done.
From https://github.com/NixOS/nixpkgs
   9314327deef..dfd115a1169  master     -> refs/nix-review/0
$ git worktree add /home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/nixpkgs dfd115a116913a6ca9c2058856029c754833073b
Preparing worktree (detached HEAD dfd115a1169)
Updating files: 100% (20699/20699), done.
HEAD is now at dfd115a1169 swiftclient: add setuptools
$ nix-env -f /home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit 2c5c0029d1702ddfaef9829bd58929618e57667b
Automatic merge went well; stopped before committing as requested
$ nix-env -f /home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/nixpkgs -qaP --xml --out-path --show-trace --meta
$ nix build --no-link --keep-going --max-jobs 4 --option build-use-sandbox true -f /home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/build.nix
[1 built, 0.0 MiB DL]
2 package were built:
dodgy python38Packages.dodgy

$ nix-shell /home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/shell.nix
innovation

[nix-shell:~/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b]$ dodgy 
Unable to read '/home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/nixpkgs/pkgs/tools/archivers/zip/natspec-gentoo.patch.bz2': 'utf-8' codec can't decode byte 0xd4 in position 10: invalid continuation byte
Unable to read '/home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/nixpkgs/pkgs/tools/X11/xbindkeys-config/xbindkeys-config-patch1.patch': 'utf-8' codec can't decode byte 0xe9 in position 312: invalid continuation byte
Unable to read '/home/kamado/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b/nixpkgs/pkgs/games/rili/moderinze_cpp.patch': 'utf-8' codec can't decode byte 0xe9 in position 3689: invalid continuation byte
{
  "warnings": [
    {
      "path": "nixpkgs/pkgs/servers/mail/mailman/settings.py",
      "line": 43,
      "code": "secret",
      "message": "Possible hardcoded secret key"
    }
  ]
}

[nix-shell:~/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b]$ dodgy --help
usage: dodgy [-h] [--ignore-paths IGNORE_PATH [IGNORE_PATH ...]] [--zero-exit]

A very basic tool to run against your codebase to search for "dodgy" looking
values. It is a series of simple regular expressions designed to detect things
such as accidental SCM diff checkins, or passwords/secret keys hardcoded into
files.

optional arguments:
  -h, --help            show this help message and exit
  --ignore-paths IGNORE_PATH [IGNORE_PATH ...], -i IGNORE_PATH [IGNORE_PATH ...]
                        Paths to ignore
  --zero-exit, -0       Dodgy will exit with a code of 1 if problems are
                        found. This flag ensures that it always returns with 0
                        unless an exception is raised.

[nix-shell:~/.cache/nix-review/rev-2c5c0029d1702ddfaef9829bd58929618e57667b]$ exit
$ git worktree prune
kamado:~/Documents/nixpkgs$ git push --force
Enumerating objects: 17, done.
Counting objects: 100% (17/17), done.
Delta compression using up to 4 threads
Compressing objects: 100% (9/9), done.
Writing objects: 100% (10/10), 1.85 KiB | 1.85 MiB/s, done.
Total 10 (delta 7), reused 0 (delta 0)
remote: Resolving deltas: 100% (7/7), completed with 7 local objects.
To github.com:kamadorueda/nixpkgs.git
 + f4e8dbce456...2c5c0029d17 pythonPackages.dodgy -> pythonPackages.dodgy (forced update)
kamado:~/Documents/nixpkgs$

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

diff LGTM
has test 👍

[3 built, 6 copied (2.4 MiB), 0.6 MiB DL]
https://github.com/NixOS/nixpkgs/pull/77255
2 package were built:
dodgy python38Packages.dodgy

@jonringer
Copy link
Contributor

@GrahamcOfBorg build dodgy python38Packages.dodgy

@jonringer jonringer merged commit f467c3e into NixOS:master Jan 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants