Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir #78495

Closed
wants to merge 1 commit into from

Conversation

bhipple
Copy link
Contributor

@bhipple bhipple commented Jan 26, 2020

This has several advantages:

  1. It takes up less space on disk in-between builds in the nix store.
  2. It uses less space in the binary cache for vendor derivation packages.
  3. It uses less network traffic downloading from the binary cache.
  4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only
    substitute --flat hashes on single files (not recursive directory hashes).
  5. It's consistent with how simple fetchurl src derivations work.
  6. It provides a stronger abstraction between input src-package and output
    package, e.g., it's harder to accidentally depend on the src derivation at
    runtime by referencing something like ${src}/etc/index.html. Likewise, in
    the store it's harder to get confused with something that is just there as a
    build-time dependency vs. a runtime dependency, since the build-time
    src dependencies are tarred up.

Disadvantages are:

  1. It takes slightly longer to untar at the start of a build.

As currently implemented, this attaches the compacted vendor.tar.gz feature as a
rider on verifyCargoDeps, since both of them are relatively newly implemented
behavior that change the cargoSha256.

If this PR is accepted, I will push forward the remaining rust packages with a
series of treewide PRs to update the cargoSha256s.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

…e hash dir

This has several advantages:

1. It takes up less space on disk in-between builds in the nix store.
2. It uses less space in the binary cache for vendor derivation packages.
3. It uses less network traffic downloading from the binary cache.
4. It plays nicely with hashed mirrors like tarballs.nixos.org, which only
   substitute --flat hashes on single files (not recursive directory hashes).
5. It's consistent with how simple `fetchurl` src derivations work.
6. It provides a stronger abstraction between input src-package and output
   package, e.g., it's harder to accidentally depend on the src derivation at
   runtime by referencing something like `${src}/etc/index.html`. Likewise, in
   the store it's harder to get confused with something that is just there as a
   build-time dependency vs. a runtime dependency, since the build-time
   src dependencies are tarred up.

Disadvantages are:
1. It takes slightly longer to untar at the start of a build.

As currently implemented, this attaches the compacted vendor.tar.gz feature as a
rider on `verifyCargoDeps`, since both of them are relatively newly implemented
behavior that change the `cargoSha256`.

If this PR is accepted, I will push forward the remaining rust packages with a
series of treewide PRs to update the `cargoSha256`s.
@bhipple
Copy link
Contributor Author

bhipple commented Jan 26, 2020

This works fine for the new behavior, but needs a bit more work for backwards compat on legacy behavior; will re-open when ready.

@bhipple bhipple closed this Jan 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant