Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PoC] fetchurl: use Software Heritage as a content addressable mirror #53787

Closed
wants to merge 4 commits into from
Closed

[PoC] fetchurl: use Software Heritage as a content addressable mirror #53787

wants to merge 4 commits into from

Conversation

veprbl
Copy link
Member

@veprbl veprbl commented Jan 11, 2019 •
edited

This is a proof of concept implementation of the idea from #53653.

Can be tested like this:

with import ./. {};

fetchurl {
  # simulate a dead URL (we never packaged version 1.2.9)
  url = "gopher://example.com/not_a_thing/zlib-1.2.9";

  # base32 hash as commonly used in nixpkgs
  sha256 = "08dabvnaqis3jlhyzwvh0dmz4lsq5zsmdbyjjm4fgl8yycp31avk";
}
trying https://archive.softwareheritage.org/api/1/content/sha256:73ab302ef31ed1e74895d2af56f52f5853f26b0370f3ef21954347acec5eaa21/raw/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  593k  100  593k    0     0   144k      0  0:00:04  0:00:04 --:--:--  144k
/nix/store/3ynhg01hji29wsv1g177f5sqyx6zfr91-zlib-1.2.9

To make this really work we would need to figure out how to convert hashes without "nix-hash". Also this would require some kind of negotiation with Software Heritage, and, perhaps, we would need to set up some kind of caching proxy.

Obviously, do not merge this.

@asymmetric
Copy link
Contributor

How does Guix do this?

@nlewo
Copy link
Member

nlewo commented Jul 6, 2019 •
edited

@asymmetric According to https://issues.guix.info/issue/33432, it seems they fallback to Software Heritage for their git fetcher. They can use the commit id as an identifier.
@veprbl Why using nix-hash is an issue (I was not able to find an another tool to do this actually)?

@veprbl
Copy link
Member Author

veprbl commented Jul 7, 2019

@nlewo This explains a lot. When I originally made this PR I tested it with zlib and it so happened that they had that tarball. Later I tried to locate couple other tarballs on SH to no avail. It seems like they prefer to mirror whole repositories instead, and that would be a reason for guix to implement this for git fetcher.

The problem with using nix in fetchurl is that produces a circular dependency. It should not be impossible to implement the hash conversion in some other way (e.g. write it in bash).

@nlewo
Copy link
Member

nlewo commented Jul 8, 2019

@veprbl it seems to be a pain to translate hash in bash! There are several hash types (md5, sha256, sha512) and the base32 encoding is not trivial (https://github.com/NixOS/nix/blob/master/src/libutil/hash.cc#L173)!
So, instead we could add a builtin such as hashToBase16 that takes a hash and returns its base16 representation.
But to start, I would suggest to break the cycle by ignoring the Software Heritage fallback for Nix itself. We could for instance add an extra arg enableSoftwareHeritageFallback ? true to fetchurl and override it to false for the Nix package. For all other packages, we could then use nix-hash for the hash translation.

@veprbl
Copy link
Member Author

veprbl commented Jul 9, 2019

This should just work now (at least on darwin). Conversion is only implemented for sha256 hashes for now.

@veprbl veprbl mentioned this pull request Oct 25, 2019
Closed
@veprbl veprbl linked an issue Feb 17, 2020 that may be closed by this pull request
Use Software Heritage as a fallback download location #53653
Open
@stale

This comment has been minimized.

Sign in to view
@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020
@veprbl veprbl removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 2, 2020
@domenkozar
Copy link
Member

https://www.tweag.io/blog/2020-06-18-software-heritage/

@veprbl veprbl closed this Dec 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
6.topic: fetch 6.topic: reproducible builds 9.needs: community feedback 10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use Software Heritage as a fallback download location
5 participants
@veprbl @asymmetric @nlewo @domenkozar @GrahamcOfBorg