haskell: add a callStack2nix function #61067

cdepillabout · 2019-05-07T02:20:51Z

callStack2nix is a function similar to callCabal2nix, but instead of just creating a .nix file for a single Haskell package, it creates a whole overlay based on an Stackage LTS resolver.

This makes it very easy to build a Stack-based Haskell project with Nix.

callStack2nix uses the stack2nix tool to produce a .nix file as output. You can see an example of what one of these files look like in the stack2nix repo:

https://raw.githubusercontent.com/input-output-hk/stack2nix/1143ceb8b395478d4e5bc8d2b0b0dee19e2f140a/stack2nix.nix

There is another function, callStack2nixPkgSet, that actually does the import-from-derivation so that the package set produced from callStack2nix is usable.

There is an example in the documentation, but here is a another example:

let
  src = fetchFromGitHub {
    owner = "chrisdone";
    repo = "intero";
    rev = "8da81244783fbf03afb49660423c875f2e874fba";
    sha256 = "17vibxapzp4wf0dfc56x98wsf3wy98ghj5h10nyf7xcfwy6k0rja";
  };

  interoPkgSet = haskellPackages.callStack2nixPkgSet {
    name = "intero";
    inherit src;
    sha256 = "163mmyl1c718g3i5r7djr4x2m07id190zb3pjl69flh0wyxwzzzc";
    hackageSnapshotTimestamp = "2019-04-16T00:00:00Z";
    haskellPackagesCompiler = pkgs.haskell.packages.ghc822;
    doCheck = false;
    doHaddock = false;
  };

in

interoPkgSet.intero

This example demonstrates using callStack2nixPkgSet to easily build the intero tool.

Motivation for this change

I wanted an easy way to build a stack-based Haskell project with nix.

Every once in a while it comes up on the issue tracker or IRC that someone wants an easy way to build a Stack-based Haskell project with nix, and there is currently no way to do this. I think it would be nice to have a short-and-sweet method for doing this in nixpkgs.

There are a few alternatives (like using stack2nix by hand to generate a package set, using haskell.nix, etc), but they all require more work and additional understanding to use.

Pinging @peti. I opened this as a PR against master, because it doesn't really affect any of the other Haskell stuff, but I can always rebase it against the haskell-updates branch if this is easier.

Things done

callStack2nix is a function similar to callCabal2nix, but instead of just creating a .nix file for a single Haskell package, it creates a whole overlay based on an Stackage LTS resolver. This makes it very easy to build a Stack-based Haskell project with Nix.

cdepillabout · 2019-05-07T02:37:20Z

There is a lot to dislike about this approach:

Since stack2nix uses the internet, the derivation produced by callStack2nix needs to be a fixed-output derivation. In practice, this is somewhat annoying to work with, for all the reasons that fixed-output derivations are normally annoying to work with.

It also prevents callStack2nix from be used for local development. However, there are a lot of ways to hack on stack-based project locally (buildStackProject, running stack2nix locally, haskell.nix), so I guess this isn't a deal-breaker.
IFD. callStack2nixPkgSet uses IFD to import the package set produced by callStack2nix. Using IFD is always somewhat unpleasant. At least with callStack2nix, the derivation will only have to be evaluated once. After that, the output will live in the nix store, so it will be quick to re-use.
General complexity of stack2nix. stack2nix wasn't originally meant to be run inside of nix-build, so it is somewhat hacky getting it to work. Of course, this work has already been done in this PR, so in the future people should be able to "just use it".

In general, I think that the advantages brought by callStack2nix and callStack2nixPkgSet out-weigh the disadvantages above. It is really nice to be able to build a stack-based Haskell project with just a few lines of nix code.

cdepillabout · 2019-05-07T02:41:11Z

peti, I see that you have removed yourself as a Haskell code-owner. Sorry for pinging you above!

peti · 2019-05-07T12:35:03Z

peti, I see that you have removed yourself as a Haskell code-owner. Sorry for pinging you above!

No problem. You'll probably want to ping @domenkozar, who is making the decisions about the future development of the Haskell infrastructure these days.

domenkozar · 2019-05-09T09:24:40Z

pkgs/development/haskell-modules/make-package-set.nix

+    # environment.  However, callStack2nix should not be used if you want to
+    # hack on the PureScript compiler itself.
+    #
+    # TODO: Currently this doesn't work for stack.yaml files that have extra-deps


@cdepillabout what is the unfortunate interaction? I'd be interested to using your work but need to address this restriction :)

I didn't look at this super deeply, but I think what happens is the following:

stack2nix calls into the stack codebase to do a bunch of stuff. Part of this process involves stack downloading all of the extra-deps defined in the stack.yaml. extra-deps can be packages that are up on Hackage. They can also be Haskell packages that are specified as URLs, git repos, etc. These packages are downloaded to the .stack-work/downloaded/ directory.

stack2nix then calls into the cabal2nix codebase to actually create nix derivations for each Haskell package defined in the stackage LTS, as well as all the packages defined in extra-deps.

For extra-deps that can be fetched from Hackage, I think cabal2nix just downloads the .cabal file for the package from Hackage and runs on that. For packages that can't be fetched from Hackage (for instance, Haskell packages that are only available on Github), cabal2nix downloads the repo using tools like nix-prefetch-git, and then runs on the .cabal file contained in the repo. However, using prefetchers like nix-prefetch-git doesn't work when running under nix-build.

Ideally, I'd like to get this PR merged in, since it does still work in many instances (specifically, for any stack project that doesn't contain Git repos as extra-deps).

After that, I'd like to send a PR to the stack2nix repo fixing it up so that it doesn't cause cabal2nix to try to call tools like nix-prefetch-git. I think that this can be done by calling cabal2nix on the copy of the repo downloaded to .stack-work/downloaded/, instead of passing the URL of the repo to cabal2nix as done currently.

I think a better way is to add support to all those prefetchers not to use nix-store --add as populating nix store isn't really needed.

I'm not entirely sure to merge this before that's done, as it will give a lot of failed builds and bug reports. I do see it brings value to you though, I'm a bit undecisive to be honest.

I think a better way is to add support to all those prefetchers not to use nix-store --add as populating nix store isn't really needed.

I don't know if this change would be accepted to those prefetchers, since putting stuff into /nix/store is basically why they exist.

However, I think you're on the right track. It might be a good idea to open up an issue against cabal2nix that stops it from using the prefetchers, and instead just downloads things directly with tools like git. There is no reason for it to need to use the prefetchers, aside from the fact that the prefetchers make it easy to calculate the sha256 hash. Although I don't know how easy it would be to get this change into cabal2nix. It may be simpler to just work around it in stack2nix.

I'm not entirely sure to merge this before that's done, as it will give a lot of failed builds and bug reports.

Yeah, I think I agree with you here. I think we need to determine whether we value having this (somewhat?) useful function in nixpkgs, even though we know it won't work for some people, or whether we value reducing bug reports.

When working on this PR, I searched for Haskell repos online I could test it on, and I actually couldn't find even one that used non-Hackage extra-deps (granted, I only searched for about 5 minutes). To me, this pushes it over the line of usefulness, so I'm slightly in favor of merging it in.

But I definitely agree that it would be ideal to have it working fully in all cases before merging it in.

I don't know if this change would be accepted to those prefetchers, since putting stuff into /nix/store is basically why they exist.

Maybe we could have "hash generators" that prefetchers would use, but I'm not entirely sure that it's worth the effort than just adding something like --no-store-add toggle to existing prefetchers.

I think we need to determine whether we value having this (somewhat?) useful function in nixpkgs

Maybe with revised documentation that clearly states the limitations (I had to ask a question why this TODO means), it would add value while documenting where it won't work. Bonus would be actually parsing extra-deps and check if it uses anything else than cabal package and fail with a good error. But then energy is better spent just getting fetchers to be more generic :)

Maybe we could have "hash generators" that prefetchers would use, but I'm not entirely sure that it's worth the effort than just adding something like --no-store-add toggle to existing prefetchers.

I created an issue about either having a hash generator, or adding a --no-store-add flag to the nix-prefetch-url command:

NixOS/nix#2919

I haven't updated the documentation yet on this PR, but I will let you know when I do.

@domenkozar I just added a commit that changes the documentation to explain in more detail when callStack2nix cannot be used currently:

2c40da7

I think PR should now be ready for a final review and merge.

chreekat · 2019-05-16T19:20:25Z

One passing comment on names: the name of all these functions follow the same [call]Foo2nix convention, but they produce very different things. Some produce .nix files, some produce whole packages. Plus, the call* variants often do very different things than the tool they call. It's impossible to keep them straight.

I'm not sure there's anything to be done about it in this issue, but since I was in the neighborhood, I thought I'd throw it out there.

cdepillabout · 2019-06-04T13:32:44Z

@chreekat I definitely agree with you. The callFoo2nix functions all output different things, and it is tough to know in advance what you can expect from them without reading the documentation.

If you have any suggestions on how better name the two functions I've added here (callStack2nix and callStack2nixPkgSet), please let me know.

callCabal2nix produces a derivation for building the specified package, so maybe we could change callStack2nixPkgSet to actually be callStack2nix, and then have the current callStack2nix function be called something like generateStack2nix.

cdepillabout · 2019-11-30T08:44:30Z

I think this is too hacky to really end up in nixpkgs. Also since stack2nix is not really being maintained, it doesn't make sense to ever merge this.

cdepillabout requested a review from basvandijk as a code owner May 7, 2019 02:20

ofborg bot added 6.topic: haskell 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels May 7, 2019

domenkozar reviewed May 9, 2019

View reviewed changes

cdepillabout mentioned this pull request Jun 4, 2019

add --no-store-add flag to nix-prefetch-url NixOS/nix#2919

Open

Update documentation to explain when callStack2nix cannot be used.

2c40da7

cdepillabout closed this Nov 30, 2019

cdepillabout deleted the callStack2nix branch November 1, 2021 05:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

haskell: add a callStack2nix function #61067

haskell: add a callStack2nix function #61067

cdepillabout commented May 7, 2019

cdepillabout commented May 7, 2019 •

edited

cdepillabout commented May 7, 2019

peti commented May 7, 2019

domenkozar May 9, 2019

cdepillabout May 9, 2019 •

edited

domenkozar May 9, 2019

cdepillabout May 9, 2019 •

edited

domenkozar May 13, 2019

cdepillabout Jun 4, 2019

cdepillabout Jun 4, 2019 •

edited

chreekat commented May 16, 2019 •

edited

cdepillabout commented Jun 4, 2019

cdepillabout commented Nov 30, 2019

haskell: add a callStack2nix function #61067

haskell: add a callStack2nix function #61067

Conversation

cdepillabout commented May 7, 2019

Motivation for this change

Things done

cdepillabout commented May 7, 2019 • edited

cdepillabout commented May 7, 2019

peti commented May 7, 2019

domenkozar May 9, 2019

Choose a reason for hiding this comment

cdepillabout May 9, 2019 • edited

Choose a reason for hiding this comment

domenkozar May 9, 2019

Choose a reason for hiding this comment

cdepillabout May 9, 2019 • edited

Choose a reason for hiding this comment

domenkozar May 13, 2019

Choose a reason for hiding this comment

cdepillabout Jun 4, 2019

Choose a reason for hiding this comment

cdepillabout Jun 4, 2019 • edited

Choose a reason for hiding this comment

chreekat commented May 16, 2019 • edited

cdepillabout commented Jun 4, 2019

cdepillabout commented Nov 30, 2019

cdepillabout commented May 7, 2019 •

edited

cdepillabout May 9, 2019 •

edited

cdepillabout May 9, 2019 •

edited

cdepillabout Jun 4, 2019 •

edited

chreekat commented May 16, 2019 •

edited