Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

b2sum: init at 20160619 #44459

Merged
merged 3 commits into from Aug 6, 2018
Merged

b2sum: init at 20160619 #44459

merged 3 commits into from Aug 6, 2018

Conversation

kirelagin
Copy link
Member

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@kirelagin kirelagin force-pushed the b2sum branch 2 times, most recently from a014988 to 11a9769 Compare August 4, 2018 14:57
@thoughtpolice
Copy link
Member

Note that b2sum is already part of coreutils since release 8.26

root@link> readlink $(which b2sum)
/nix/store/jd2j5s966bwhgvrj9fw1jjrxr8m0m33r-coreutils-8.29/bin/b2sum

Since this is in stdenv, I don't think there's any need for this package I'm afraid?

Copy link
Member

@lukateras lukateras left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to have this package in Nixpkgs, for two reasons:

  1. This supports algorithms other than BLAKE2b: BLAKE2s, BLAKE2bp, BLAKE2sp.
  2. While coreutils and busybox have this utility, it may be unwieldy to install the full set for people on macOS (or in the future, BSDs: none to my knowledge include BLAKE2 digest program).

{ stdenv, fetchurl, openmp }:

stdenv.mkDerivation rec {
version = "20160619";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this release, upstream has added ARM NEON implementations of BLAKE2b and BLAKE2s which should improve performance on aarch64: BLAKE2/BLAKE2@7965d3e

Also a bug fix for BLAKE2bp which previously returned incorrect results when -l option was set: BLAKE2/BLAKE2#36

To incorporate these changes into Nixpkgs version, could you use version from master?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh my, why didn’t they make a new release then :/
we should probably tell the Homebrew people

url = "https://github.com/BLAKE2/BLAKE2/archive/${version}.tar.gz";
sha256 = "0csnlp6kwlyla5s4r6bsrx2jgcwrm9qzisnvfdhmqsz5r8y87b6b";
};
postUnpack = "sourceRoot=$sourceRoot/b2sum";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sourceRoot = "b2sum";?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Um, I’m not sure it would work this way, because the source root is <whatever the dir in the archive is called>/b2sum

@@ -0,0 +1,25 @@
{ stdenv, fetchurl, openmp }:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since openmp is nullable, would be nice to have it default to null:

{ stdenv, fetchurl, openmp ? null }:

@@ -1579,6 +1579,10 @@ with pkgs;

asynk = callPackage ../tools/networking/asynk { };

b2sum = callPackage ../tools/security/b2sum {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably move tthsum from ../tools/misc to ../tools/security then, for conformance.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I judged by tthsum and rhash and decided the latter was placed more appropriately, so, yes, I think, we should.

@thoughtpolice
Copy link
Member

thoughtpolice commented Aug 4, 2018

I'm not really sure I buy argument 1 that having multiple hash functions is necessarily worth adding this package. People always think they can choose something better because of their particular know-how, but the actual users who can do this are relatively small -- to the detriment and confusion of other, more ordinary users. BLAKE2b is a perfectly fine and robust hash function for the use-cases b2sum is intended for (file hashes, more or less), and it should already be faster than MD5 or sha1sum which I'd consider unsafe, not to mention SHA2 variants:

root@link> dd if=/dev/urandom of=test bs=1024 count=$((1024*1024))
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 17.3029 s, 62.1 MB/s
root@link> cat test > /dev/null # prewarm disk cache
root@link> time md5sum test
e99888c6e9ace85a6bd5716e10355526  test
md5sum test  1.48s user 0.08s system 99% cpu 1.556 total
root@link> time b2sum test
e02d2c9dfe6b6216a5c81d155f75875c15ca51ffdcfea8ae1e984d6862f449212b4eb2e6bc8506d4162a1a254753327df7cc58510a4474b9d8887c8afb06ac7a  test
b2sum test  1.30s user 0.09s system 99% cpu 1.389 total
root@link> time sha1sum test
f449d243faec4e432e69b6d36114b9d6bf9d1407  test
sha1sum test  1.53s user 0.12s system 99% cpu 1.654 total
root@link> time sha256sum test
55d0fffd7e5b9377be427f1ef82557ab4d5a37bb3098c428a831dc51cdfcf163  test
sha256sum test  3.75s user 0.10s system 99% cpu 3.855 total

Personally, I consider "algorithm flexibility" for things like this -- that are mostly inconsequential and confusing to non-familiar/'crypto' users -- to be anti-features.

While blake2b is a 64-bit hash function, I don't think our (relatively small) 32-bit userbase should really worry about having multiple hash functions for e.g. blake2s for some efficiency gains. Having multiple hash functions also makes it unclear which one you used, and which package people should use for doing their hash comparisons (especially because any use or installation of b2sum will conflict with/override coreutils' installation), though BLAKE2b is already the default for the original b2sum as well.

There's an argument the parallel versions might be worth having, but frankly I don't find that super convincing as an argument for duplicating an already existing utility that seems perfectly adequate on its own, although I admit some people may disagree.


I also don't buy argument 2: MacOS already has coreutils in stdenv, does it not? And it comes with b2sum (I verified this by looking at Hydra closures, as I have no macOS machines). It has to work, so I'm not sure there's any problem there at all. And "future BSDs" are not a good argument for adding packages and/or increasing the footprint of nixpkgs, in my opinion (Actually, I consider Nix-on-not-Linux a misguided 'feature' that largely penalizes the vast majority of users who are on Linux by complexifying build expressions -- though I accept macOS as a valid target -- but that's neither here nor there). Besides, I suspect trying to truly remove coreutils semantics from the majority of Nix build expressions that use them via stdenv (and, by doing so, allowing coreutils, and thus b2sum to be moved out of the closure of stdenv) to be a complexity nightmare of huge proportions that would affect a huge amount of packages.

Tentatively adding features that are based on potential future users (that historically have fallen to bitrot very quickly) for relatively minor reasons seems like a bad idea, and can always be rectified in the future if needed. We can just cross that bridge later.


I generally don't (and haven't, I think) put hard rejections on package additions, but I'm rather disinclined to accept this addition, personally. I can't say that my opinion weighs more heavily than anyone elses, though (I'm just an average maintainer).

@lukateras
Copy link
Member

I understand your arguments and agree with them for the most part, just not your conclusion.

Nixpkgs has always been very friendly to new packages, we regularly accept packages with virtually no users, or packages that have better counterparts. The policy so far was to let the end-user make the decision, and just be a collection of build instructions with nearly no moderation.

Since this is a different implementation of b2sum, one that is created by BLAKE2 upstream, and is different in its capabilities (algorithms) and dependencies from the rest, I don't think we can decline it without reconsidering acceptance policy at larger scale.

@kirelagin
Copy link
Member Author

kirelagin commented Aug 4, 2018

algorithm flexibility

I don’t think that the “crypto algorithm flexibility is bad” argument can be used in this case. This sort of tools most of the time is used for hash verification, not generation and when a user has to verify a hash, they have no choice which algorithm to use. In this specific case it is not an issue at all, simply because the tool does not explicitly ask the user to pick an algorithm, it has the right one as the default.

coreutils in stdenv

The fact that coreutils is in stdenv doesn’t mean that it is in the user’s profile. If I need to compute b2sum frequently and want it in my path, you are essentially suggesting that I replace all my core command line tools with the ones from Linux (or use some tricks to add only one binary from coreutils to path).

Let me explain how this PR happened. I needed to compute a blake2 hash of a file, so I check that shasum that I have doesn’t support it, googled shasum blake and all the links I found were to this implementation. Obviously, I became happy that there exists not just some third-party utility, but an actual reference implementation, so naturally I decided I will use it, so I checked whether it was in nixpkgs and created an expression myself. I can easily imagine other users following the same path. I should say that when looking for this tool I couldn’t even imagine that it was reimplemented in coreutils and I have to make a confession: even now, when I know that it is there, I have really hard time trying to imagine a reason for reimplementing it in coreutils instead of using this small reference implementation.

Update: I have looked up the discussion in the mailing list and apparently the reason was “we already have md5sum and it is bad”. And it is not a reimplementation, it is the exact same code. In my opinion, the fact that the developers of coreutils decided to include a copy of some tool in their package is irrelevant to the question of inclusion of this tool in nixpkgs.

@lukateras lukateras merged commit 0d7170c into NixOS:master Aug 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants