Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

amass: init at 2.8.3 #50490

Merged
merged 1 commit into from Nov 18, 2018
Merged

amass: init at 2.8.3 #50490

merged 1 commit into from Nov 18, 2018

Conversation

kalbasit
Copy link
Member

@kalbasit kalbasit commented Nov 17, 2018 •
edited

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: amass

Partial log (click to expand)

shrinking /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin/bin/amass.netdomains
shrinking /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin/bin/amass
shrinking /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin/bin/amass.db
shrinking /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin/bin/amass.maltego
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin/bin
patching script interpreter paths in /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin
checking for references to /build in /nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin...
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
/nix/store/zsd8l0pf1nrr16vl4k6l17778ysmxgyr-amass-2.8.0-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: amass

Partial log (click to expand)

shrinking /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin/bin/amass.maltego
shrinking /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin/bin/amass.netdomains
shrinking /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin/bin/amass
shrinking /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin/bin/amass.db
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin/bin
patching script interpreter paths in /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin
checking for references to /build in /nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin...
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
/nix/store/5x70y9gd9x75mnlyfldh8540ab8rnfxa-amass-2.8.0-bin

@c0bw3b
Copy link
Contributor

c0bw3b commented Nov 17, 2018

There seems to be v2.8.3 already on https://github.com/OWASP/Amass/releases
It's not properly tagged on GH but seems like a stable release (already on BlackArch)

@kalbasit
Copy link
Member Author

@c0bw3b yes I saw that but v2.8.3 does not compile so I went with the latest release instead of the latest tag.

@kalbasit
Copy link
Member Author

@c0bw3b I took a deeper look and it just had a left-over import because the project moved from @caffix to @OWASP, PTAL.

@kalbasit kalbasit changed the title amass: init at 2.8.0 amass: init at 2.8.3 Nov 18, 2018
@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: amass

Partial log (click to expand)

shrinking /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin/bin/amass
shrinking /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin/bin/amass.viz
shrinking /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin/bin/amass.db
shrinking /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin/bin/amass.netdomains
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin/bin
patching script interpreter paths in /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin
checking for references to /build in /nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin...
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
/nix/store/qdhl67g5jc7r7y0rgr7fsis5cjdcby19-amass-2.8.3-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: amass

Partial log (click to expand)

shrinking /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin/bin/amass.netdomains
shrinking /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin/bin/amass.viz
shrinking /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin/bin/amass
shrinking /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin/bin/amass.db
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin/bin
patching script interpreter paths in /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin
checking for references to /build in /nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin...
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
/nix/store/61jmnxd5pagaxvfci2fa6xqqsp7pzr6n-amass-2.8.3-bin

@c0bw3b c0bw3b self-assigned this Nov 18, 2018
c0bw3b
c0bw3b approved these changes Nov 18, 2018
View reviewed changes
Copy link
Contributor

@c0bw3b c0bw3b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

$ ./result-bin/bin/amass -version
version 2.8.3

$ ./result-bin/bin/amass -v -ip -brute -min-for-recursive 3 -d somedomain.tld
[...]
OWASP Amass v2.8.3                                https://github.com/OWASP/Amass
--------------------------------------------------------------------------------
44 names discovered - api: 15, scrape: 11, brute: 1, dns: 2, cert: 15
--------------------------------------------------------------------------------

Running great. I wondered if we should ship the wordlists with it but it would increase the closure size significantly and by default if -brute is passed the tool will just use https://raw.githubusercontent.com/OWASP/Amass/master/wordlists/namelist.txt

@c0bw3b c0bw3b merged commit 9a9925e into NixOS:master Nov 18, 2018
@kalbasit kalbasit deleted the nixpkgs_add-amass branch November 19, 2018 04:28
@kalbasit
Copy link
Member Author

@c0bw3b thx for merging! w.r.t the wordlist, we could write a wrapper around the binary to download the wordlist from master and caches it by the commit hash somewhere in XDG_CACHE? That way we always get the freshest set and the closure size would not increase by more than the wrapper itself.

@c0bw3b
Copy link
Contributor

c0bw3b commented Nov 19, 2018

Humm XDG_CACHE_HOME is user-specific and we want to share an identical set of wordlists among users. There is XDG_DATA_DIRS maybe...
But the XDG spec is not ubiquitous and not cross-platform (not followed on Darwin or Windows FWIW).
This is how restic deals with this for example: restic/restic#1287

The nix way would really be to install it into $out/share/amass/
Or maybe make a multi-output package and install it into $doc or something...

@caffix
Copy link

caffix commented Nov 19, 2018

@c0bw3b Perhaps we can bring that wordlist into the source code

@c0bw3b
Copy link
Contributor

c0bw3b commented Nov 19, 2018 •
edited

Oh hey! nice to see the original author here :) (btw nice tool)

For you upstream I guess it's good to keep the wordlists as separate data. Plus it allows distro packagers to choose whether they want to ship it or not (or just a subset of it).
Our main issue for nixpkgs/nixos is not that it is separate, but more the size it adds if we decide to ship it along with the tool.

Another random idea: ship the wordlists compressed (.txt.gz) and zcat them at runtime?
I don't know if it's even a practical idea in Go. Don't mind me if it's too far-fetched. :p

@kalbasit
Copy link
Member Author

@c0bw3b I like the idea of shipping the wordlist gzipped but we will have to send a patch upstream (or maybe @caffix can help us with it) to first un-gzip the file if it ends with .gz here.

@c0bw3b
Copy link
Contributor

c0bw3b commented Nov 19, 2018

Yes I was thinking reading the compressed text file directly at runtime from the Go code.
Maybe with https://golang.org/pkg/compress/gzip/ or https://github.com/AlasdairF/Custom ?

@kalbasit
Copy link
Member Author

kalbasit commented Nov 19, 2018 •
edited

compress/gzip is the way to go, it can be done with an io.Pipe to improve performance. It's pretty straightforward, I'll submit a PR tonight if @caffix hasn't gotten to it by then.

EDIT: since we are only reading the file (no writing) a pipe is not actually necessary.

@caffix
Copy link

caffix commented Nov 20, 2018

@kalbasit I'd be happy to receive your contribution. Thank you!

@kalbasit kalbasit mentioned this pull request Nov 20, 2018
Merged
@kalbasit
Copy link
Member Author

@caffix thx for merging owasp-amass/amass#56!

@c0bw3b should I simply put the wordlist under $out/share and have the user explicitly find where it is or should I implement a wrapper to always use the wordlist regardless?

@c0bw3b
Copy link
Contributor

c0bw3b commented Nov 20, 2018

There is multiple wordlists and I think we should let the user decides when to use one, and which one.
So just install in $out\share

You can explain this in meta.longDescription so users could be aware of it

@kalbasit kalbasit mentioned this pull request Nov 20, 2018
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@c0bw3b c0bw3b c0bw3b approved these changes

Assignees

@c0bw3b c0bw3b

Labels
8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@kalbasit @GrahamcOfBorg @c0bw3b @caffix