haskell generic-builder: Use strictDeps always #41420

Ericson2314 · 2018-06-03T16:55:26Z

Motivation for this change

This helps avoid the ARG_MAX issues we've been having, and is generally a good idea to ensure cross comparability anyways.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

peti · 2018-06-03T17:50:49Z

https://hydra.nixos.org/eval/1460999 is a build of the Haskell package set with this change applied.

colonelpanic8 · 2018-06-03T17:57:29Z

@peti @Ericson2314 Is there anything blocking merge here?

peti · 2018-06-03T18:28:01Z

Yes, the test builds are not complete yet.

colonelpanic8 · 2018-06-03T22:50:44Z

@Ericson2314 It seems like strictDeps does not ameliorate this issue when it is encountered with stack builds (or maybe I'm doing something totally incorrectly). Do you have any idea how to fix the stack build of taffybar?

colonelpanic8 · 2018-06-03T23:13:07Z

@Ericson2314 Actually, it seems that what happens when you enable crossConfig/stictDeps with the stack builder is that all of the haskell-gi generated bindings fail with

setup: The program 'pkg-config' version >=0.9.0 is required but it could not

I suspect that this is probably because the paths that are passed to the stack build are not properly inherited in the builds for each haskell package, but I have to say that I'm pretty out of my depth with this haskell/stack/nix stuff.

peti

This change breaks haskell.packages.ghc843.hspec-discover. See the "new failing jobs" tab at https://hydra.nixos.org/eval/1460999 for further details.

Ericson2314 · 2018-06-05T02:38:40Z

Weird! This is because GHC cannot find the testsuite from the same package to use a pre-processor. We have tests for this in Cabal https://github.com/haskell/cabal/tree/master/cabal-testsuite/PackageTests/BuildToolDepends; I'd know as I wrote them!

Ericson2314 · 2018-06-19T04:16:49Z

https://hydra.nixos.org/eval/1464500

Ericson2314 · 2018-06-19T04:21:35Z

CC @kirelagin @angerman @matthewbauer

Ericson2314 · 2018-06-19T14:47:32Z

Looks like there's some lingering issues with build tools: hspec-discover despite my patch, and maybealex too. If anyone has any ideas I'd love to hear them.

Ericson2314 · 2018-06-19T15:06:38Z

Rebased on #42247 which should work to keep things moving.

Ericson2314 · 2018-06-22T17:13:07Z

https://hydra.nixos.org/eval/1465142. It will probably fail again, but I'd love to be wrong!

matthewbauer · 2018-06-22T19:24:50Z

pkgs/development/haskell-modules/configuration-common.nix

-  taffybar = super.taffybar.overrideDerivation (drv: {
-    strictDeps = true;
+  # https://github.com/hspec/hspec/pull/355
+  hspec-discover_2_5_1 = appendPatch super.hspec-discover_2_5_1 (pkgs.fetchpatch {


I think this needs to be done for hspec-core (and maybe other hspec stuff) as well:

ghc: could not execute: hspec-meta-discover

OK I'll try

Ah ok fixed. didn't realize this was the executable from hspec-meta, not hspec-discover.

Ericson2314 · 2018-06-23T00:42:12Z

OK fixed this build, reevaluating. We'll see what fails.

Ericson2314 · 2018-06-23T18:59:25Z

Huh. My evaluation hydra evaluation is stuck pending, and just disappeared before that.

Needed for new ‘strictDeps = true;’ handling. Including- - slim - string-conversions - skeletal-test - catamorphism - unliftio - logging-facade - distributive - doctest - http-types - interpolate - mockery - mime-mail - http2

Weird errors otherwise - seems to need this to know it can skip the bad test.

This is needed for picking up hspec-discover in strictDeps generic-builder.nix.

domenkozar · 2018-07-03T08:49:21Z

This change requires a lot of manual overrides. I think solution @angerman is proposing is less intrusive. Another way is to actually fix the commit that made MAX_ARGS explode: https://github.com/NixOS/nixpkgs/pull/41939/files#diff-360bed088faac7580cee137eeffb39e0R279

Ericson2314 · 2018-07-03T15:40:47Z

@domenkozar the big change motivating all the overrides here is we're being explicit for the first time about when we need a binary from another package or just its library. So I think these changes will be very good for multiple outputs (#32629). Also, Cabal 3.0 due at the end of the year will remove the non-new-build stuff, in which case being explicit in the Cabal file will also be required.

My guess is Cabal will need to come up with some sort of back-compat story, and we can replace most the overrides with that, probably implemented in Cabal2nix but who knows.

domenkozar · 2018-07-03T16:05:24Z

Oh you mean stuff like this: sol/with-location#1

OK, but why is this needed in order to unbreak ARG_MAX? :)

domenkozar · 2018-07-03T21:57:56Z

Could someone pull in domenkozar@4149778

These have changed on master.

matthewbauer · 2018-07-03T23:29:31Z

Merging for today because of the many breakages on macOS. There are still breakages for now but we should be below 400. strictDeps seems to be the only way to fix this without also breaking cross compilation of haskell.

domenkozar · 2018-07-04T09:12:12Z

@matthewbauer btw adding build tool dependencies should go to configuration-nix as it's not version dependent :) Thanks!

domenkozar · 2018-07-04T18:15:43Z

FYI: it was reverted and I agree with @peti we can autogenerate these: NixOS/cabal2nix#356

…h not be non-empty. This commit was originally introduced as part of NixOS#41420 and then reverted with the rest of that PR. However there was no reason to revert his particular commit.

Ericson2314 requested a review from peti as a code owner June 3, 2018 16:55

Ericson2314 mentioned this pull request Jun 3, 2018

taffybar: work around repeated arguments bug #41408

Merged

8 tasks

Ericson2314 force-pushed the haskell-strict-deps branch from 6e0311a to b4c2a16 Compare June 3, 2018 16:57

GrahamcOfBorg added 6.topic: haskell 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Jun 3, 2018

puffnfresh approved these changes Jun 3, 2018

View reviewed changes

peti requested changes Jun 4, 2018

View reviewed changes

Ericson2314 force-pushed the haskell-strict-deps branch from 09e0c1d to da046de Compare June 19, 2018 02:29

Ericson2314 force-pushed the haskell-strict-deps branch from da046de to 8f353c8 Compare June 19, 2018 15:01

Ericson2314 force-pushed the haskell-strict-deps branch from 8f353c8 to 73ca88d Compare June 22, 2018 17:02

GrahamcOfBorg added the 2.status: merge conflict label Jun 22, 2018

Ericson2314 requested a review from matthewbauer June 22, 2018 17:13

matthewbauer reviewed Jun 22, 2018

View reviewed changes

Ericson2314 force-pushed the haskell-strict-deps branch from 73ca88d to a413506 Compare June 23, 2018 17:06

GrahamcOfBorg added 8.has: clean-up and removed 2.status: merge conflict labels Jun 23, 2018

matthewbauer and others added 10 commits July 2, 2018 15:52

text-conversions: add hspec-discover

160894d

haskell: add more test tool depend for hspec-discover

fea9a71

Needed for new ‘strictDeps = true;’ handling. Including- - slim - string-conversions - skeletal-test - catamorphism - unliftio - logging-facade - distributive - doctest - http-types - interpolate - mockery - mime-mail - http2

haskell with-location: Link upstreaming attempt

c4d4376

with-location: Link upstreamming attempt

3457ce0

haskell-modules: Fix mockery in macos

42bd49d

Weird errors otherwise - seems to need this to know it can skip the bad test.

haskell-modules: Add many more overrides for hspec-discover

10f1063

This is needed for picking up hspec-discover in strictDeps generic-builder.nix.

haskell-modules: Add test tool depend for modern-uri

c4431d3

haskell-modules: Add pandoc for heist

fb94cd9

haskell-modules: Add more test tool depends

2cb5633

haskell-modules: Add more overrides

4590a37

matthewbauer force-pushed the haskell-strict-deps branch from 9abf866 to 4590a37 Compare July 2, 2018 19:56

GrahamcOfBorg removed the 2.status: merge conflict label Jul 2, 2018

yesod-persistent: add hspec-discover for 1.4.3

1a668f7

domenkozar and others added 2 commits July 3, 2018 19:24

configuration-common.nix: fix build tools

9ed54a3

configuration-ghc-8.4.x.nix: Update hspec versions

157b597

These have changed on master.

GrahamcOfBorg removed the 10.rebuild-darwin-stdenv label Jul 3, 2018

matthewbauer force-pushed the haskell-strict-deps branch from 53596e2 to 157b597 Compare July 3, 2018 23:26

matthewbauer merged commit f682ff9 into NixOS:master Jul 3, 2018

matthewbauer deleted the haskell-strict-deps branch July 3, 2018 23:32

peti mentioned this pull request Jul 4, 2018

Revert "haskell generic-builder: Use strictDeps always" #43016

Merged

Ericson2314 mentioned this pull request Jul 4, 2018

Migrating the ecosystem to use build-tool-depends haskell/cabal#5412

Open

Ericson2314 mentioned this pull request Jul 15, 2018

fetchpatch: quote excludes #43538

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

haskell generic-builder: Use strictDeps always #41420

haskell generic-builder: Use strictDeps always #41420

Ericson2314 commented Jun 3, 2018 •

edited

peti commented Jun 3, 2018

colonelpanic8 commented Jun 3, 2018

peti commented Jun 3, 2018

colonelpanic8 commented Jun 3, 2018

colonelpanic8 commented Jun 3, 2018

peti left a comment

Ericson2314 commented Jun 5, 2018

Ericson2314 commented Jun 19, 2018

Ericson2314 commented Jun 19, 2018 •

edited

Ericson2314 commented Jun 19, 2018

Ericson2314 commented Jun 19, 2018

Ericson2314 commented Jun 22, 2018

matthewbauer Jun 22, 2018

Ericson2314 Jun 22, 2018

Ericson2314 Jun 23, 2018

Ericson2314 commented Jun 23, 2018

Ericson2314 commented Jun 23, 2018

domenkozar commented Jul 3, 2018

Ericson2314 commented Jul 3, 2018

domenkozar commented Jul 3, 2018

domenkozar commented Jul 3, 2018

matthewbauer commented Jul 3, 2018

domenkozar commented Jul 4, 2018

domenkozar commented Jul 4, 2018

haskell generic-builder: Use strictDeps always #41420

haskell generic-builder: Use strictDeps always #41420

Conversation

Ericson2314 commented Jun 3, 2018 • edited

Motivation for this change

Things done

peti commented Jun 3, 2018

colonelpanic8 commented Jun 3, 2018

peti commented Jun 3, 2018

colonelpanic8 commented Jun 3, 2018

colonelpanic8 commented Jun 3, 2018

peti left a comment

Choose a reason for hiding this comment

Ericson2314 commented Jun 5, 2018

Ericson2314 commented Jun 19, 2018

Ericson2314 commented Jun 19, 2018 • edited

Ericson2314 commented Jun 19, 2018

Ericson2314 commented Jun 19, 2018

Ericson2314 commented Jun 22, 2018

matthewbauer Jun 22, 2018

Choose a reason for hiding this comment

Ericson2314 Jun 22, 2018

Choose a reason for hiding this comment

Ericson2314 Jun 23, 2018

Choose a reason for hiding this comment

Ericson2314 commented Jun 23, 2018

Ericson2314 commented Jun 23, 2018

domenkozar commented Jul 3, 2018

Ericson2314 commented Jul 3, 2018

domenkozar commented Jul 3, 2018

domenkozar commented Jul 3, 2018

matthewbauer commented Jul 3, 2018

domenkozar commented Jul 4, 2018

domenkozar commented Jul 4, 2018

Ericson2314 commented Jun 3, 2018 •

edited

Ericson2314 commented Jun 19, 2018 •

edited