Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "fetchzip and friends: Set "name" to "source" by default" #31004

Closed
wants to merge 1 commit into from

Conversation

orivej
Copy link
Contributor

@orivej orivej commented Oct 30, 2017

c3255fe#commitcomment-25286020

This change invalidated all fetch* hashes, and nothing in staging could be built.

NixOS@c3255fe#commitcomment-25286020

This change invalidated all fetch* hashes, and nothing could be built.
@orivej orivej requested a review from edolstra October 30, 2017 21:12
orivej referenced this pull request Oct 30, 2017
This makes them produce the same store paths as builtins.fetchgit,
builtins.fetchTarball etc. See
NixOS/nix@65b5f17.
@vcunat
Copy link
Member

vcunat commented Oct 30, 2017

/cc @edolstra.

@domenkozar
Copy link
Member

I don't understand why this should be reverted, isn't that what staging is for?

@vcunat
Copy link
Member

vcunat commented Oct 30, 2017

@domenkozar: the claim is that it would require us to rewrite all the hashes in sources.

@orivej
Copy link
Contributor Author

orivej commented Oct 30, 2017

@domenkozar The value of sha256 in fetchurl and all other fetch functions depends on name (which by default is computed from the URL). So if you change the name, you have to recomputesha256, or nix-build will exit with e.g.

output path _/nix/store/arsizl1pa5ch8d6wgh49x8vvbfb9qyz1-source_ has r:sha256 hash _03wd1qvkrj50fjszb2apzdkc8d5bpfbbi9pajl0vbrlzzmmi3jlq_ when _0636absdfjx8ybglwydmqxwfwmqz1c4b9s5mhxlgm4ci18lw3hms_ was expected

That commit changed names of all fetch derivations without updating their hashes.

@edolstra
Copy link
Member

This is incorrect. The output hash does not depend on name. The store path does, but changing the store path should not be a problem.

For example, building nix-repl.src with name set to foo or bar respectively gives store paths:

/nix/store/gbxjginw6h789qp5xk9m788zqrgc43ms-foo
/nix/store/hbz2r8m1czcsfjg6yvn27263zgks3gwm-bar

without requiring a change to the outputHash.

@edolstra
Copy link
Member

Do you have an example of a particular package that fails to build?

@LnL7
Copy link
Member

LnL7 commented Oct 31, 2017

@orivej I suspect you'll also get the same hash mismatch if you force a rebuild of the sources.
This sometimes happens if upstream changed something in the tarball or when the hash was updated incorrectly at some point.

nix-build -A hello.src --check

@orivej orivej closed this Oct 31, 2017
@vcunat
Copy link
Member

vcunat commented Oct 31, 2017

I think Hydra will discover the failing fetches soon on staging, and we'll have to fix those individual hashes. Probably hunspell and lua failures from here: https://hydra.nixos.org/eval/1406150?compare=1406145#tabs-now-fail

@orivej orivej mentioned this pull request Oct 31, 2017
8 tasks
@orivej
Copy link
Contributor Author

orivej commented Oct 31, 2017

@LnL7 Indeed. The rebuild failed with hash mismatch, I knew that the hash in the nix store depends on derivation name, but I did not consider that it may be computed from the specified sha256 value rather than actual contents.

@edolstra

Do you have an example of a particular package that fails to build?

The 0636abs… mismatch is from luaPackages.luastdlib.src. Apparently release.zip is a link to the latest GitHub release. I've fixed it in #31031.


Still, the motivation for c3255fe is not clear to me. How is that useful? Readable names in nix store are definitely desirable (e.g. a860360) — I would not like to turn /nix/store/hash-name/ into /nix/store/hash/ neither for all derivations nor just for sources.

@edolstra
Copy link
Member

@orivej For the motivation, see Domen's NixCon talk. We currently have 5 different names/store paths generated for the same tree (in builtins.fetchTarball, builtins.fetchGit, Nixpkgs nix-prefetch-git, Hydra nix-prefetch-git, or using ./.). Thus, for example, the same tree built via builtins.fetchGit ./. will produce a different store path than when built by Hydra. This patch is part of making them the same.

@vcunat
Copy link
Member

vcunat commented Nov 1, 2017

The original commit is now in master, though some fallback hasn't been resolved yet. Some of it are just uncovering of force-updated git tags upstream and similar stuff, some are packages assuming that the name of the source has a particular format – with some special cases where you have multiple sources and thus get a new name collision during unpacking (example: dmd).

Having the less descriptive name is a disadvantage. Perhaps in future we could devise a nicer scheme for default naming that's also consistent among all the fetching mechanisms...

@orivej
Copy link
Contributor Author

orivej commented Nov 8, 2017

For the motivation, see Domen's NixCon talk.

For the record, Domen's talk is here and the slides are here (slide 8).

@copumpkin
Copy link
Member

For what it's worth, this does actually have a nontrivial effect on some derivations. Most notably for me are image builds, which actually care about the store path in question to inject a channel into the image being built. Not a big deal, but it was confusing for a bit 😄

@Hodapp87
Copy link
Contributor

Hodapp87 commented Nov 19, 2017

c3255fe seems to break pkgs.python35Packages.tensorflowWithCuda; haven't investigated yet, but filed #31828

@orivej orivej deleted the revert-fetch-name branch December 15, 2017 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants