lib: add fake hashes #53754

danbst · 2019-01-10T13:27:58Z

Etalon fake hashes can be used as placeholders for all the places, where
Nix expression requires a hash, but we don't yet have one.

This should replace all current ways to do that:

echo|sha256sum, copy into clipboard, go to editor, paste into previously
edited place
search nixpkgs for a random package, copy it's hash to cliboard, go to
editor, paste into previously edited place

Nix can add support for etalon fake hashes. In that case printed error should contain
only 1 hash, so no more problem "which of two hashes from error should I use?"

Idea by irc:Synthetica

grahamc · 2019-01-10T13:30:36Z

lib/trivial.nix

+  # Generated with `echo|XXXSum`
+  fakeMd5 = "68b329da9893e34099c7d8ad5cb9c940";
+  fakeSha256 = "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b";
+  fakeSha512 = "be688838ca8686e5c90689bf2ab585cef1137c999b48c70b92f67a5c34dc15697b5d11c982ed6d71be1e1e7f7b4e0733884aa97c3f7a339a8ed03577cf74be09";


How about just 0s?

Synthetica9 · 2019-01-10T13:31:03Z

Maybe make them more recognisable in output? Something like all 0's or all 1's, so that you know that it isn't the hash you're looking for. (Even better would be a specialised error message that doesn't mention the place-holder, but that would probably be non-trivial to implement)

grahamc · 2019-01-10T13:32:12Z

lib/trivial.nix

@@ -294,4 +294,11 @@ rec {
  */
  isFunction = f: builtins.isFunction f ||
    (f ? __functor && isFunction (f.__functor f));
+
+  # Etalon fake hashes. Can be used as hash placeholders, when computing hash


What is "Etalon"?

grahamc · 2019-01-10T13:36:11Z

lib/trivial.nix

@@ -295,8 +295,7 @@ rec {
  isFunction = f: builtins.isFunction f ||
    (f ? __functor && isFunction (f.__functor f));

-  # Etalon fake hashes. Can be used as hash placeholders, when computing hash


French words were fine, but ok :)

danbst · 2019-01-10T13:36:16Z

thanks @grahamc @Synthetica9 , removed french word and replaced with 0s. Much better!

grahamc · 2019-01-10T13:36:48Z

This looks great. Can you add something to https://nixos.org/nixpkgs/manual/#chap-quick-start about this, maybe instead of nix-prefetch-* ?

7c6f434c · 2019-01-10T13:40:22Z

Does this change our stance about certificate verification in fetchers?

(not sure if this is more discoverable than 64a0 or its non-Vim equivalent…)

Synthetica9 · 2019-01-10T13:43:34Z

@7c6f434c for 64a0 to work, you need to remember the number of characters that each hash type takes.

danbst · 2019-01-10T14:25:34Z

@7c6f434c

Does this change our stance about certificate verification in fetchers?

can you explain a bit?

edolstra · 2019-01-10T14:41:26Z

@7c6f434c That's not really affected by this PR, since it's already an issue whenever you do a nix-build with a fake hash.

7c6f434c · 2019-01-10T14:47:33Z

@danbst Right now fetchurl source says «and work on SSL without a certificate (this isn't a security problem because we check the cryptographic hash of the output anyway)»; nix-prefetch-url does verify TLS certificate, and builtins.fetchurl does unless hash is specified.

(By the way, nix-prefetch-url supports '<nixpkgs> -A something.src which I got reminded about by checking Nix source of nix-prefetch-url — it is described in the Nix reference but not mentioned in the Nixpkgs manual)

@edolstra The idea of documenting fake hashes as the primary way of obtaining the correct hash is a bit of a last straw here.

@Synthetica9 Yes, but SHA-256 is 256 bits i.e. 64 4-bit nibbles, SHA-512 is 512 bits i.e. 128 nibbles, and MD5 is either something shorter (i.e. 32 nibbles, which is correct) or a reason to reconsider the idea of adding a new MD5-hashed entry…

Synthetica9 · 2019-01-10T15:00:30Z

@edolstra @7c6f434c https://github.com/NixOS/ofborg/blob/released/factoids.toml#L10 ?

grahamc · 2019-01-10T15:09:37Z

lib/trivial.nix

@@ -294,4 +294,9 @@ rec {
  */
  isFunction = f: builtins.isFunction f ||
    (f ? __functor && isFunction (f.__functor f));
+
+  # Fake hashes. Can be used as hash placeholders, when computing hash ahead isn't trivial
+  fakeMd5 = "00000000000000000000000000000000";


Let's delete the md5 option. People shouldn't use it anymore.

danbst · 2019-01-10T15:29:54Z

I agree with @7c6f434c , this shouldn't be "official". It also doesn't belong to #chap-quick-start, newcomers shouldn't learn hacks from start. The advice to use nix-prefetch-url is perfectly fine there.

@grahamc we can spread word on wiki though. It's more targeted for "useful hacks".

edolstra · 2019-01-10T15:40:50Z

I'm not sure whether lib is the right place for useful hacks though. (Maybe lib/deprecated.nix?)

danbst · 2019-01-10T17:23:45Z

@edolstra done

@7c6f434c @grahamc
I think what misses in manual is how various ways of getting a hash interfere with security concerns.
I know some people didn't like that Nix hash has to be different from upstream one in some cases. If we point out all security issues, then we can include lib.fakeSha256 as an example. I'll create a separate doc PR

Synthetica9 · 2019-01-10T17:26:27Z

lib/trivial.nix

@@ -294,4 +294,5 @@ rec {
  */
  isFunction = f: builtins.isFunction f ||
    (f ? __functor && isFunction (f.__functor f));
+


Probably should not touch this file if it's going in misc

Fake hashes can be used as placeholders for all the places, where Nix expression requires a hash, but we don't yet have one. This should be more convenient than following: - echo|sha256sum, copy into clipboard, go to editor, paste into previously edited place - search nixpkgs for a random package, copy it's hash to cliboard, go to editor, paste into previously edited place Nix can add support for these fake hashes. In that case printed error should contain only 1 hash, so no more problem "which of two hashes from error should I use?" Idea by irc:Synthetica

joepie91 · 2019-01-14T09:51:05Z

I agree with @7c6f434c , this shouldn't be "official". It also doesn't belong to #chap-quick-start, newcomers shouldn't learn hacks from start. The advice to use nix-prefetch-url is perfectly fine there.

So what is the correct way to do this, and where is it documented? And is it actually more ergonomic than this "hack"?

danbst · 2019-01-14T11:12:51Z

@joepie91 #53785

joepie91 · 2019-01-14T11:24:41Z

Thanks. I feel like there should probably be a more streamlined utility for this purpose, though... one that can deal with all source variants automatically in a single command. Right now, using an all-zeroes hash is still the easiest "just works" method :/

7c6f434c · 2019-01-14T13:46:32Z

I think I am generally for fetch* doing certificate validation by default with an option to switch off, and maybe a global config option to switch that off.

Any drawbacks?

danbst · 2019-01-14T17:58:40Z

Don't see any. I imagine only a situation, when system hadn't been updated for ages, and all certificates invalidated. Trying to upgrade stuff in this scenario may not work (but should work with current solution).

Even if it is done, not every url is https, so fake-hash method usage still is insecure when applied blindly.

7c6f434c · 2019-01-14T20:05:09Z

Don't see any. I imagine only a situation, when system hadn't been updated for ages, and all certificates invalidated. Trying to upgrade stuff in this scenario may not work (but should work with current solution).

This is why a global switch to skip validation is useful; but I am not sure anything beyond a config option is needed for this case (it's not like we want to encourage this use case…)

Even if it is done, not every url is `https`, so fake-hash method usage still is insecure when applied blindly.

Well, even nix-prefetch-url on https URLs doesn't catch all the most likely attacks (starting with a server compromise). And with http nothing convenient can do even as much verification. I don't think we can solve anything cheaply (a Nixpkgs verification service checking upstream tarball consistency across time and source networks would be cool, but complicated, and still wouldn't be enough, in a sense), but maybe switching verification on is a cheap (small) net improvement.

danbst requested review from edolstra and nbp as code owners January 10, 2019 13:27

grahamc reviewed Jan 10, 2019

View reviewed changes

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jan 10, 2019

grahamc reviewed Jan 10, 2019

View reviewed changes

danbst force-pushed the lib-fake-hashes branch from 04c8405 to 3642c3d Compare January 10, 2019 17:12

Synthetica9 requested changes Jan 10, 2019

View reviewed changes

danbst force-pushed the lib-fake-hashes branch from 3642c3d to 68a6b47 Compare January 10, 2019 17:27

Mic92 changed the title ~~lib: add étalon fake hashes~~ lib: add fake hashes Jan 10, 2019

Mic92 merged commit b75aff7 into NixOS:master Jan 10, 2019

danbst deleted the lib-fake-hashes branch January 10, 2019 18:02

danbst mentioned this pull request Jan 11, 2019

manual: document ways of obtaining source hashes #53785

Merged

worldofpeace mentioned this pull request Sep 21, 2020

Create a lib/derivation-utils.nix (and move fake hashes there) #98422

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lib: add fake hashes #53754

lib: add fake hashes #53754

danbst commented Jan 10, 2019

grahamc Jan 10, 2019

Synthetica9 commented Jan 10, 2019 •

edited

grahamc Jan 10, 2019

grahamc Jan 10, 2019

danbst commented Jan 10, 2019

grahamc commented Jan 10, 2019

7c6f434c commented Jan 10, 2019

Synthetica9 commented Jan 10, 2019

danbst commented Jan 10, 2019

edolstra commented Jan 10, 2019

7c6f434c commented Jan 10, 2019

Synthetica9 commented Jan 10, 2019

grahamc Jan 10, 2019

danbst commented Jan 10, 2019

edolstra commented Jan 10, 2019

danbst commented Jan 10, 2019

Synthetica9 Jan 10, 2019

joepie91 commented Jan 14, 2019

danbst commented Jan 14, 2019

joepie91 commented Jan 14, 2019

7c6f434c commented Jan 14, 2019

danbst commented Jan 14, 2019

7c6f434c commented Jan 14, 2019 via email

lib: add fake hashes #53754

lib: add fake hashes #53754

Conversation

danbst commented Jan 10, 2019

grahamc Jan 10, 2019

Choose a reason for hiding this comment

Synthetica9 commented Jan 10, 2019 • edited

grahamc Jan 10, 2019

Choose a reason for hiding this comment

grahamc Jan 10, 2019

Choose a reason for hiding this comment

danbst commented Jan 10, 2019

grahamc commented Jan 10, 2019

7c6f434c commented Jan 10, 2019

Synthetica9 commented Jan 10, 2019

danbst commented Jan 10, 2019

edolstra commented Jan 10, 2019

7c6f434c commented Jan 10, 2019

Synthetica9 commented Jan 10, 2019

grahamc Jan 10, 2019

Choose a reason for hiding this comment

danbst commented Jan 10, 2019

edolstra commented Jan 10, 2019

danbst commented Jan 10, 2019

Synthetica9 Jan 10, 2019

Choose a reason for hiding this comment

joepie91 commented Jan 14, 2019

danbst commented Jan 14, 2019

joepie91 commented Jan 14, 2019

7c6f434c commented Jan 14, 2019

danbst commented Jan 14, 2019

7c6f434c commented Jan 14, 2019 via email

Synthetica9 commented Jan 10, 2019 •

edited