Skip to content

haskellSrc2nix: disallow substitutes #49829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

guibou
Copy link
Contributor

@guibou guibou commented Nov 6, 2018

Fix #49818. See the bug for more details.

Motivation for this change

The work done by haskellSrc2nix is usually lightweight, but needs to
be done each time a cabal file change. This triggers unexpected
network activity for a cache query which results most of the time on a
cache miss.

By disabling substitutes we:

  • avoid the, mostly useless, binary cache query
  • avoid unexpected network failure when users change a cabal file and
    try to rebuild without connectivity.
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

Sorry, something went wrong.

Fix NixOS#49818. See the bug for more details.

The work done by haskellSrc2nix is usually lightweight, but needs to
be done each time a cabal file change. This triggers unexpected
network activity for a cache query which results most of the time on a
cache miss.

By disabling substitutes we:

- avoid the, mostly useless, binary cache query
- avoid unexpected network failure when users change a cabal file and
  try to rebuild without connectivity.
@GrahamcOfBorg GrahamcOfBorg added 6.topic: haskell 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Nov 6, 2018
@srhb
Copy link
Contributor

srhb commented Nov 13, 2018

cc @basvandijk in case the codeowners ping gets lost in the storm. :-)

@lheckemann
Copy link
Member

Shouldn't this already be covered by preferLocalBuild? According to the manual:

If this attribute is set to true, it has two effects. First, the derivation will always be built, not substituted, even if a substitute is available. Second, if distributed building is enabled, then, if possible, the derivaton will be built locally instead of forwarded to a remote machine. This is appropriate for trivial builders where the cost of doing a download or remote build would exceed the cost of building locally.

@srhb
Copy link
Contributor

srhb commented Nov 14, 2018

@lheckemann The manual was misleading, but has been corrected: NixOS/nix@5a3f140 NixOS/nix@1109193

@lheckemann
Copy link
Member

Aah, I see. Thanks!

@Profpatsch Profpatsch merged commit 62882d8 into NixOS:master Nov 22, 2018
@puffnfresh
Copy link
Member

Good change! Thank you

@akshaymankar
Copy link
Contributor

Is there a way to override this change easily?
Context: My team has a fairly big project and we a few forked dependencies and a few pinned dependencies. In all running callCabal2nix over and over again on them takes about 3 minutes in our CI. These would almost always be a cache hit for us.

@guibou guibou deleted the guibou-haskellsrc2nix-disallow-substitutes branch October 17, 2022 15:59
@guibou
Copy link
Contributor Author

guibou commented Oct 17, 2022

@akshaymankar you can just override all your callCabal2nix with allowSubstitutes = true;, this could be done in a overlay, something like:

selfh: superh: {
    callCabal2nix = name: path: args: (superh.callCabal2nix name path args).overrideAttrs(_old: { allowSubstitutes = true; });
}

However, I'm curious you are raising this question right now. Maybe it is related to #194753 that I'm trying to merge now?

@akshaymankar
Copy link
Contributor

Thanks for the pointer, I ended up with this:

hself: hsuper: {
    haskellSrc2nix = args: (hsuper.haskellSrc2nix args).overrideAttrs(_old: {allowSubstitutes = true;});
}

But in all I gained almost no time for about 40 packages, so I removed it.

However, I'm curious you are raising this question right now. Maybe it is related to #194753 that I'm trying to merge now?

I never had to pin so many packages in any of my other projects, so I never even noticed that substitutes were not allowed for this derivation. So, it is not related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: haskell 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux
Projects
None yet
Development

Successfully merging this pull request may close these issues.

haskellSrc2nix should not use substituters
7 participants