Hash always has a valid type #3738

Ericson2314 · 2020-06-23T18:25:03Z

@edolstra based on your passing remark on our call, we made some of the ValidPathInfo/NarInfo hashes std::optional, but we're not really sure what the correct "bussiness logic" is, so that will be important to review.

~~These two remaining test failures may also turn up interesting things.~~ edit nope, they weren't.

The parser changes use some of the same techniques I used in #3730 -- lots of std::string_view including a rest I mutate as I go----in general trying to write the code as if I had proper parser combinators at my disposal.

The rest is rote.

… into hash-always-has-type

Ericson2314 · 2020-06-24T14:12:19Z

src/libstore/store-api.hh

@@ -115,7 +115,8 @@ struct ValidPathInfo
 {
    StorePath path;
    std::optional<StorePath> deriver;
-    Hash narHash;
+    // TODO document this
+    std::optional<Hash> narHash;


Here is one of the now-optional hashes

@edolstra So I checked and while this one is optional, the constructor for NarInfo (derived class) requires that it is present with assertions. Is that good enough for now, or do you also want the refactor so that it is required everywhere? (I do recall you were saying the nix installer trick where the nar hash would be missing is long gone.)

(I must admit due to #3640 I am a bit partial to keeping it optional, but I'm happy to do whatever you like.)

I think this one should not be optional since a store path should always have a NAR hash.

This is actually turns out to be a fairly substantial change, because it looks like the BinaryCacheStore::addToStore overload that doesn't take the ValidPathInfo was uploading flat files as-is and calling them nars, with the narHash unset.

I think that a bug not intended feature, but either way, fixing it is the sort of major behavior change I rather do separately from this refactor which has no behavior changes?

Ericson2314 · 2020-06-24T14:12:53Z

src/libfetchers/tree-info.hh

@@ -11,7 +11,7 @@ namespace nix::fetchers {

 struct TreeInfo
 {
-    Hash narHash;
+    std::optional<Hash> narHash;


Though I think this one is fine and just means the hash is calculted---obviously these builtins aren't going to be making non-CA paths.

I think this one should not be optional. Anyway it's better to leave libfetchers unchanged since it's all rewritten in the flakes branch.

We thought it might be easy to fix this, but it isn't either: the caching stuff doesn't store the narHash and since the cache also doesn't distinguish between stores, doing a queryPathInfo to get the Nar hash in the case of a cache-hit doesn't work reliably enough to pass the test suite.

Perhaps something could be made to work, but like you said libfetchers is all rewritten anyways, and I think the 2 lines we changed in libfetchers with the std::optional is much more minimal than whatever it would take to make it work without the std::optional.

This change did go away with the flakes branch being merged, yay!

Ericson2314 · 2020-06-24T14:16:08Z

src/libstore/nar-info.hh

@@ -10,7 +10,7 @@ struct NarInfo : ValidPathInfo
 {
    std::string url;
    std::string compression;
-    Hash fileHash;
+    std::optional<Hash> fileHash;


Per meeting this one is OK

The aim of this check was just to ensure each key occurs once.

edolstra · 2020-07-27T09:00:16Z

There are some conflicts, could you have a look at those? Otherwise LGTM.

Ericson2314 · 2020-07-27T14:24:27Z

OK, fixed! Will get the PRs that include this too.

Ericson2314 · 2020-07-27T15:55:55Z

Again?! :D

Ericson2314 · 2020-07-27T16:16:28Z

Fixed. Guess I shouldn't complain as it's my own other PR which caused the conflict :)

B4dM4n · 2020-07-30T11:19:41Z

This PR introduced a evaluation error for me when building the niv package:

% nix run 'nix/86805a2c0a25f5ceefac0d64e64ba57ace73b7f5' -- build -f https://github.com/nmattia/niv/archive/c0a61e6283933fa6eed6d3451ffd86c9df5209ec.tar.gz -v --show-trace
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
store path mismatch in (possibly filtered) path added from '/nix/store/p46a908qp9pg5qdvm0v20kwisqixjdcq-source'
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- show-trace ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
trace: while evaluating 'sourceByRegex'
at: (8:30) in file: /nix/store/p46a908qp9pg5qdvm0v20kwisqixjdcq-source/default.nix

     7|
     8|   sourceByRegex = name: src: regexes:
      |                              ^
     9|     builtins.path {

trace: from call site
at: (22:16) in file: /nix/store/p46a908qp9pg5qdvm0v20kwisqixjdcq-source/default.nix

    21|
    22|   niv-source = sourceByRegex "niv" ./. [
      |                ^
    23|     "^package.yaml$"

The same build was successful with a5f7d31 (one commit before):

nix run 'nix/a5f7d310dd10fe86b6f6aa1c2771c30f113741d4' -- build -f https://github.com/nmattia/niv/archive/c0a61e6283933fa6eed6d3451ffd86c9df5209ec.tar.gz -v --show-trace

Not sure if niv is using builtins.path in a wrong way (default.nix#L9) and it needs to be fixed on their side.

Ericson2314 · 2020-07-30T15:15:18Z

Thanks for reporting this, I do think it's a bug on the Nix side I'm sorry to have introduced. #3845 is the next step of this process, and I think will root out any renaming bugs, but if that can't be done fast enough we can separately investigate this one

B4dM4n · 2020-07-31T11:08:36Z

The build failure is fixed in #3881. Thanks for your effort.

Ericson2314 · 2020-07-31T14:58:06Z

Actually was totally coincidental! (As far as I know.) Thanks, Matt!

Ericson2314 and others added 7 commits June 19, 2020 19:32

WIP: Make Hash always store a valid hash type

20799a5

WIP bug fixing

e7a1411

WIP: Make Hash always store a valid hash type

507aa48

More bug fixing

73ac003

Merge branch 'hash-always-has-type' of github.com:obsidiansystems/nix…

3dc10f7

… into hash-always-has-type

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

98e5d1a

Fix and clean up hash parser

ec3a857

Ericson2314 marked this pull request as draft June 23, 2020 18:42

Fix test suite

8d51d38

Ericson2314 changed the title ~~WIP: Hash always has a valid type~~ Hash always has a valid type Jun 23, 2020

Ericson2314 marked this pull request as ready for review June 23, 2020 21:42

Ericson2314 commented Jun 24, 2020

View reviewed changes

Ericson2314 added 2 commits June 29, 2020 18:03

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

987a4a0

Fix Narinfo corruption detection bug

1be279a

The aim of this check was just to ensure each key occurs once.

Ericson2314 force-pushed the hash-always-has-type branch from 058a156 to 1be279a Compare July 2, 2020 21:46

Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type

dbffd30

Ericson2314 changed the title ~~Hash always has a valid type~~ WIP: Hash always has a valid type Jul 7, 2020

meditans force-pushed the hash-always-has-type branch from afe35d6 to dbffd30 Compare July 7, 2020 15:58

Ericson2314 changed the title ~~WIP: Hash always has a valid type~~ Hash always has a valid type Jul 7, 2020

Ericson2314 and others added 3 commits July 13, 2020 03:01

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

4415765

Merge branch 'master' of github.com:NixOS/nix into hash-always-has-type

d090562

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

5ea817d

Ericson2314 mentioned this pull request Jul 21, 2020

Make ValidPathInfo::narHash mandatory #3845

Closed

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

699fc89

Merge remote-tracking branch 'upstream/master' into hash-always-has-type

43f2bd8

edolstra merged commit 86805a2 into NixOS:master Jul 27, 2020

Ericson2314 deleted the hash-always-has-type branch July 27, 2020 17:33

ldesgoui mentioned this pull request Aug 1, 2021

Write Nix 2.4 release notes #4521

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hash always has a valid type #3738

Hash always has a valid type #3738

Ericson2314 commented Jun 23, 2020 •

edited

Ericson2314 Jun 24, 2020

Ericson2314 Jun 29, 2020

edolstra Jul 7, 2020

Ericson2314 Jul 7, 2020

Ericson2314 Jun 24, 2020

Ericson2314 Jun 24, 2020

edolstra Jul 7, 2020

Ericson2314 Jul 7, 2020 •

edited

Ericson2314 Jul 16, 2020

Ericson2314 Jun 24, 2020

Ericson2314 Jun 29, 2020

edolstra commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

B4dM4n commented Jul 30, 2020

Ericson2314 commented Jul 30, 2020

B4dM4n commented Jul 31, 2020

Ericson2314 commented Jul 31, 2020

Hash always has a valid type #3738

Hash always has a valid type #3738

Conversation

Ericson2314 commented Jun 23, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Jul 7, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edolstra commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

B4dM4n commented Jul 30, 2020

Ericson2314 commented Jul 30, 2020

B4dM4n commented Jul 31, 2020

Ericson2314 commented Jul 31, 2020

Ericson2314 commented Jun 23, 2020 •

edited

Ericson2314 Jul 7, 2020 •

edited