Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package hexd and pixd #31567

Closed
wants to merge 3 commits into from
Closed

Conversation

FireyFly
Copy link
Contributor

Motivation for this change

Packaging hexd and pixd, two closely related utilities I wrote.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
    • (shouldn't be applicable I believe)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
    • (no other packages should be affected since I'm introducing two new ones, so this should be vacuously true, but feels cheeky to check this box)
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

hexd is a simple hexdump tool which colourises different ranges of bytes with different colours, which makes it easy to spot patterns in the data. It also provides some useful flags, such as the ability to only hexdump a specific range of a file (or of several files).

pixd is based on hexd but tweaked to use half-block characters and 24-bit colours, displaying each byte as a coloured half-block. Each of the 256 byte values gets assigned to a unique colour taken from a colour palette (which is overridable with an environment variable), again for purposes of showing the structure and repeating patterns. That said, in practice I find hexd to be a lot more useful for reverse-engineering purposes, personally.

I wrote pixd this spring, and although it was mostly an experiment in visualizing arbitrary binary data, a lot of people seemed to like it and it got packaged in some other assorted distros (mostly ones related to computer security, penetration testing and so on). I thought I might as well write my own nix derivations for them, and having written them I thought I might as well contribute them back as well. :)

That said, I've been using NixOS for a grand total of a week now, and these are the second and third nix derivations I've written, so it'd be great if they were thoroughly reviewed and I'd be happy for any suggestions of improvements/changes. (The first one I wrote was me packaging pangoterm, and although I'm intending to pull-request it too, it's not quite ready for that yet. Since hexd and pixd basically amount to make && make install I figured they're good newbie derivations for me too.)

@FireyFly FireyFly changed the title Pkgs/hexd and pixd Package hexd and pixd Nov 12, 2017
@bjornfor
Copy link
Contributor

Cool!

Can I suggest that you

  1. Add yourself to lib/maintainers.nix in a separate commit (just force push the rebased branch)
  2. git tag / release your projects and update the Nix files to refer to the tag/version instead of an (arbitrary) commit.

@FireyFly
Copy link
Contributor Author

Alright, will do. How should I format the commit message of the commit that only adds me to maintainers.nix? (Amusingly, I had three commits at first, but then rebased after reading the contributions guidelines, and their notes about commit message formats)

@bjornfor
Copy link
Contributor

Hehe, yes, I guess there is no standard for the lib/maintainers.nix file. (Looking at "git log -- lib/maintainers.nix" shows all kinds of formats.) I guess I'd use "maintainers: add X" or "Add X as maintainer". Your choice :-)

@FireyFly
Copy link
Contributor Author

Hmm, I made some changes to use the (newly released 😄) 1.0.0 releases of the two tools, but kind-of expected I might have to change the hashes too... however, when I try to build the derivations after making changes to them, I simply get nix-build printing the path to the old derivation's output... even though the version changed, and the path has the old version in it!

I'm not entirely sure how to properly rebuild the derivations, what am I missing here? (For future reference/my own learning, if nothing else!)

% nix-build -A hexd
/nix/store/2q8wasr5zxn1cfp9jpgrfy3nd8clrcvq-hexd-2017-11-12
% grep 'version =' pkgs/tools/misc/hexd/default.nix 
  version = "1.0.0";

@bjornfor
Copy link
Contributor

Did you remember to update the derivation name/version, which in turn makes a change in the url passed to fetchurl?

If you did, then it's a bug. If you didn't, it's a feature. (Oh, wait. You're using fetchFromGitHub... Well, it's just a wrapper around fetchurl.)

fetchurl re-downloads if the name or sha256 changes. Most often "name" is not passed directly to fetchurl, instead it takes "name" from the basename of the URL.

If there is no change in inputs for fetchurl, it returns the cached artifact (which is uniquely identified by its hash).

I mostly change the first few chars of the hash to "00000...." and rebuild the package. Then update the hash according to the error message. There is also a tool called nix-prefetch-url which can be used.

@FireyFly
Copy link
Contributor Author

FireyFly commented Nov 12, 2017

I did update the version in the derivation (that's why I put the grep line there in the shell excerpt, showing it's different from the one in the path, which is a date). Actually... okay, I'm really confused now. Changing the hash to an obviously incorrect one:

firefly@as ~/local/repos/nixpkgs
% vim pkgs/tools/misc/hexd/default.nix  # updating the hash to a dummy one
firefly@as ~/local/repos/nixpkgs
% nix-build -A hexd  # still prints the path to an old, cached build in the store!
/nix/store/2q8wasr5zxn1cfp9jpgrfy3nd8clrcvq-hexd-2017-11-12
firefly@as ~/local/repos/nixpkgs
% git diff
diff --git a/pkgs/tools/misc/hexd/default.nix b/pkgs/tools/misc/hexd/default.nix
index b080e23b..a3de5f62 100644
--- a/pkgs/tools/misc/hexd/default.nix
+++ b/pkgs/tools/misc/hexd/default.nix
@@ -10,7 +10,7 @@ stdenv.mkDerivation rec {
     owner = "FireyFly";
     repo = "hexd";
     rev = "v${version}";
-    sha256 = "1lm0mj5c71id5kpqar8n44023s1kymb3q45nsz0hjh9v7p8libp0";
+    sha256 = "zzzzzzzzzzzzzzzzzzzzz4023s1kymb3q45nsz0hjh9v7p8libp0";
   };
 
   makeFlags = [ "PREFIX=$(out)" ];

I have no clue why the old, cached build is used here, now...

I tried building with nix-build -v -A hexd, and don't even see my derivation being evaluated, somehow... I redirected stderr of the above and the same for xxd, and this is how they differ:

% diff /tmp/xxd.build.err /tmp/hexd.build.err 
34,36d33
< evaluating file ‘/home/firefly/local/repos/nixpkgs/pkgs/tools/misc/xxd/default.nix’
< evaluating file ‘/home/firefly/local/repos/nixpkgs/pkgs/applications/editors/vim/default.nix’
< evaluating file ‘/home/firefly/local/repos/nixpkgs/pkgs/applications/editors/vim/common.nix’
95,96c92
< evaluating file ‘/home/firefly/local/repos/nixpkgs/pkgs/development/libraries/ncurses/default.nix’
< derivation is /nix/store/za3d6w6fi4w4pnkfl29mpbg8kvwzyicf-xxd-8.0.1257.drv
---
> derivation is /nix/store/qjwb50la23smwvmdb681yk90pk85586p-hexd-2017-11-12.drv

I'm expecting my nixpkgs/pkgs/tools/misc/hexd/default.nix to be evaluated here, surely? Or am I missing something silly here?

@bjornfor
Copy link
Contributor

There is one odd thing here. Your derivation still outputs "/nix/store/...-hexd-2017-11-12". I think that should have been "...hexc-1.0.0", if you updated the version attribute.

The only reason I can think of causing fetchurl to return the old source package is if the basename of the url did not change.

@orivej
Copy link
Contributor

orivej commented Nov 12, 2017

This is a bit confusing, but it is intentional. c3255fe was made with the purpose to make all sources with the same content to be considered the same — in the case of fetchFromGitHub regardless of the owner, repo, and rev. Nix will not redownload your source and will not rebuild your derivation unless you change the hash. (Or it will rebuild your derivation from the old source if you change the derivation without changing source hash.) You may use nix-prefetch-url '<nixpkgs>' -A hexd.src to redownload and recompute it.

@bjornfor
Copy link
Contributor

@orivej: Thanks!

@FireyFly
Copy link
Contributor Author

@orivej Hmm, nix-prefetch-url does appear to download and print the hash.. however, changing the hash to a dummy value (all 'z', i.e. sha256 = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz") seems to still just print the cached build's store path. I'm expecting a hash mismatch error in that case, surely?

@orivej
Copy link
Contributor

orivej commented Nov 12, 2017

Could you attach the full /tmp/hexd.build.err?

@FireyFly
Copy link
Contributor Author

FireyFly commented Nov 12, 2017

hexd.build.err.log

This is stderr of nix-build -v -A hexd, issued after changing the hash in the derivation to all z's.

Edit: oh, with -vv I see this at the end:

...
instantiated ‘unzip-6.0’ -> ‘/nix/store/fs0pqs584ka5hj2bfb915zy2bdmd4nxl-unzip-6.0.drv’
instantiated ‘source’ -> ‘/nix/store/hkyfbnzd2kjalq2bdqj81pvazpx1vg0j-source.drv’
instantiated ‘hexd-2017-11-12’ -> ‘/nix/store/qjwb50la23smwvmdb681yk90pk85586p-hexd-2017-11-12.drv’
derivation is /nix/store/qjwb50la23smwvmdb681yk90pk85586p-hexd-2017-11-12.drv
/nix/store/2q8wasr5zxn1cfp9jpgrfy3nd8clrcvq-hexd-2017-11-12

Here's the full log with -vv, for good mesaure: hexd.build-vv.err.log

@orivej
Copy link
Contributor

orivej commented Nov 12, 2017

And what is the output of nix-instantiate --eval -A hexd.meta.position?

@FireyFly
Copy link
Contributor Author

FireyFly commented Nov 12, 2017

... oh. Oh dear, that was silly of me. Before deciding to put these packages in nixpkgs I was experimenting with declarative management of user packages, and had put my hexd and pixd packages inline in my ~/.config/nixpkgs/config.nix in packageOverrides--and apparently promptly forgot about that when I started moving them out to nixpkgs.

Sorry about that. I'm getting the expected hash conflict now with the all-z hash, and getting the expected output after changing to the correct hash. Thanks for helping me debug this :)


Also, I've now built the most recent versions (after switching to using the version tags) and they do build and work properly with the most recent force-pushed rebase.

@bjornfor
Copy link
Contributor

Applied to master (bdce932 and 2 parents). Thanks!

@bjornfor bjornfor closed this Nov 12, 2017
@FireyFly FireyFly deleted the pkgs/hexd-and-pixd branch May 10, 2020 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants