@hyperfekt Thanks for working on this.

There are frequently contributors that ask if there is an easy way to regenerate the hackage package set, normally for testing changes they have made.

Here is the most recent: #86659 (comment)

I think having something like this in nixpkgs would be nice, mostly for the people like the above link.

I have some specific questions about this PR though, so I'll ask those questions inline.

The update-hackage-packages.sh update-cabal2nix-latest.sh script now depends on a version of cabal2nix with NixOS/cabal2nix#455 merged.

Changed base to master to prevent diff pollution from force-pushes on haskell-updates.

Let me put it this way:
I fully understand that you derive no benefit from this PR being merged.
The PR is not meant to benefit you directly.
It is meant to benefit everyone else - its goal is to make it easy for contributors to test their changes to configuration-hackage2nix.yaml, and for people to make local changes to it that are never upstreamed because they are unlikely to serve others, or that are needed more quickly than the regeneration of hackage-packages.nix performed by you.
Ultimately my hope is that this leads to an increased number of contributors to the nixpkgs Haskell package set and with them an increased quality of the package set - ideally in a virtuous cycle with an increased number of users.

It has been shown in the past that, while certainly entirely possible, the hurdle of generating the Haskell package set anew with the current process is one that many do not clear, and there is no good reason for it being remotely this burdensome, requiring every user to replicate a frankly (for Nix standards) arcane setup; when with the power of Nix it can be as simple as executing a single command. I also recall hearing about people who have gained the understanding of how to do it still finding it too troublesome to perform.

I have chosen to include these scripts in nixpkgs because

• the required hackage2nix being served from the cache further lowers the friction of contributing, and with a prebuilt hackage2nix the cabal2nix repository would have to be fetched just for these scripts,
• I see a very low chance of this kind of portable script which is not tailored to your usecase actually being merged in the cabal2nix repository, and
• I am convinced scripts that the nixpkgs contributors use to operate on nixpkgs should also live in nixpkgs.

Your declaration of not maintaining them I take absolutely no issue with (I'm happy to do that), and your not wanting to use them I have accepted. I don't think either of these are reasons not to include them. Again, they are not for you.

I do however need your minimal support in pinning the versions used to generate hackage-packages.nix in nixpkgs. Users will often work off of older versions either because they cannot trivially upgrade, or because they need the cache provided by the NixOS foundation.

I'm not sure I see the harm in adding this sort of pinning to the scripts you use, so maybe you could enlighten me about the problems it brings instead of outright rejecting it without giving a reason.

PS: I very much appreciate your intent to make the package generation more efficient through the direct use of Cabal's tarballs. But until that is implemented I think these scripts are very useful, and even when implemented we would want (slimmer, simpler) scripts as well as pinning in order to not increase the friction to more than necessary.

@peti I have thought about this a bit more. There is one reason, why we maybe want more reproducibility than we have right now.
When someone makes a change to the yaml-file, they should probably regenerate the hackage-packages.nix, so that they can

1. test their change and
2. contribute a PR which already works and can be tested easily.

But now if everyone where to generate the hackage-packages.nix with the e.g. cabal tarball available to them at that time, this will very often produce changes at different points in hackage-packages.nix. This has multiple disadvantages:

1. It is harder to see if one accidentally changed anything else. That might be a problem for the person creating the PR and for the one reviewing it. It could even introduce evaluation errors that confuse the contributer (or at least prevent easy testing) without anything to do with the change being made.
2. In general if everyone were to regenerate hackage-packages.nix with some version of hackage, PRs would get confusing, might create much more rebase problems and a lot more rebuilds (possibly even on the contributers machine) that don‘t have anything to do with the actual change.
3. I don‘t want to think about the mess, when someone accidentally uses a slightly too old hackage version and rolls back some changes without realizing …

So I think the case can be made, that pinning the hackage version to generate hackage-packages.nix would make the contribution process much smoother. Because contributers could update exactly the part they changed, PRs would have a cleaner separation, would be easier to test and (Bonus!) situations where the PR causes a problem, when you regenerate the hackage-packages.nix, i.e. problems you need to solve, get rarer.

So I think the over all goal should be: a) Make the regeneration of the nix file easy and b) make the regeneration with the same hackage version easy.

Now, I have said nothing about what the best way to achieve that would be. But this PR supplies a solution that works right now. Another approach would be to have hackage2nix print the hackage snapshot to the top of the nix file (I know it‘s written in the commit message but that is harder to extract for tooling) and then have some tool (probably in the hackage2nix repo) that can pull that exact hackage version, be it a git-repo or a tarball …

Looking at make-package-set.nix, I think this PR currently breaks callHackage, which assumes all-cabal-hashes to be a tarball. The most direct solution is to revert it to be a tarball, and unpack it only when regenerating the package set. As an added benefit I think I can save a lot of time on slow filesystems by using the tarball via ratarmount. In the long term the direct use of the cabal tarball that peti suggested would probably be most appropriate.

There is one reason, why we maybe want more reproducibility than we have right now.

Just to avoid misunderstandings: we have reproducibility. hackage2nix records the version of itself and of Hackage that was used to generate the hackage-packages.nix file. What don't have is a --re-generate flag to update-nixpkgs.sh that resets the Hackage snapshot to the version that was used to generate the current Nixpkgs checkout, so that users can easily take advantage of that information.

I have tried to verify your claim. I remain unconvinced as of now:

NixOS/cabal2nix#459

It is likely that the issue is my mistake. But I needed 30 minutes to get to that point. I think the Haskell tooling in nixpkgs is great. I am really amazed by it. But I think it can be made easier to use.

I want to mention a script I hacked together which does at least some of this. @peti merged it into the cabal2nix repo.

NixOS/cabal2nix#461

what's blocking this ? as @hyperfekt mentioned, some haskell fixes are probably never contributed upstream because it's too demanding (I know I keep my overrides local for instance). Having these simple scripts would make it easier. Can't we pin the versions ? we don't have to run the latest of everything at all times.
It does not need to be perfect right now but once there are more users, it will improve.

@teto The issue here is that this PR, if I understand it correctly, introduces a new workflow to update the hackage packages in nixpkgs. While there is some agreement that the current situation is not optimal. This PR would need more discussion and changes to the infrastructure than just merging it. If we would just merge it, that would be very confusing because the regenerate script will not work unless something or someone continually bumps the pinned hash.

Just for the meantime: Regenerating the haskell-packages.nix based on the last hackage snapshot used in nixpkgs is a matter of 60 seconds:

1. Clone git@github.com:nixos/caba2nix into <repo>
2. Symlink your nixpkgs checkout to <repo>/nixpkgs
3. Run <repo>/regenerate-hackage.sh.

Sorry, I answered your first question. Regarding the other points:

• I think fixing stuff in nixpkgs is not very hard per-se and these scripts in my opinion would only change the difficulty slightly. I think the main issue is a lack of documentation.
  Just contact us/open an issue/start a WIP PR, we are glad to help you and it's actually not that hard.

• Re pinning versions: This PR is not about pinning versions of individual packages. We in general don‘t just run the latest of everything but we use the last stable stackage snapshot in general. On the other hand pulling new versions from hackage (for everything not in stackage and for packages in stackage if you use the explicitly versioned package name) every night. Is one of the great appeals of nixpkgs and I wouldn‘t want to miss it. I don‘t see the disadvantages either. If people don‘t want that at all, they can just use nixpkgs-stable.

contrast #86699 (comment) with running:
pkgs/misc/vim-plugins/update.py.
Even knowing the steps, I doubt its 60 seconds, and finding what to run is what takes time, understanding what parameters mean etc.
NixOS/cabal2nix#461 is a step in a good direction. I just want to confirm that we are all ok to have a script run from within nixpkgs to ease discoverability (and also because it's a guarantee it works without extra-knowledge).

I just want to confirm that we are all ok to have a script run from within nixpkgs to ease discoverability (and also because it's a guarantee it works without extra-knowledge).

Um, have you read any of the previous discussion? This is pretty much the main point of contention.

I think the main problem here is, that there is no way to make a change to a Haskell package in nixpkgs without the "extra-knowledge" you are talking about.
Scripts as documentation are a bad solution for a problem that is not solvable without understanding at least parts of it.
So I think the real issue here is documentation, not ease of use.

That being said I personally am not opposed to having an update script in nixpkgs. Maybe a very small wrapper script that just pulls cabal2nix and refers to a regenerate script in there might be a good compromise @peti ?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-ghc-default-version-8-10-2-package-versions-now-from-stackage-nightly/10117/29

Thanks to everyone who has worked on improving the hackage tooling. I just had the following experience: I wanted to use pandoc-crossref from master. I noticed it was broken, so I wanted to see why it was broken and try to fix it. I noticed that I was able to build and use pandoc-crossref just fine with NIXPKGS_ALLOW_BROKEN=1, so I wanted to remove the broken flag.

@peti as the main person working on this, what do you think I should do in this case: Should I regenerate the package set with the versions mentioned in the last update commit? Or should I just commit the hackage2nix configuration and wait for it to be picked up on an automated update. What would I have to be careful about when committing changes to the package set?

Scripts as documentation are a bad solution for a problem that is not solvable without understanding at least parts of it.
So I think the real issue here is documentation, not ease of use.

I don't fully agree. I think I was able to get an idea about how all the tools and repositories work together quite quickly, however it's still more work to regenerate the package set in practise. For example, if I had found the regenerate-nixpkgs.sh script by @maralorn earlier that would have saved me a lot of time. I looked in these places (in this order):

• in pkgs/development/haskell-modules/: nothing, no regenerate script, no README
• in the Nixpkgs manual: There's a link to haskell4nix.readthedocs.io.
• haskell4nix.readthedocs.io: I thought I found something, the haskell4nix Nixpkgs developers guide, but it's essentially an empty page other than a few links to videos, which is a nice addition but not a replacement for a helper script since it takes a lot more time to follow a video and reproduce the steps in comparison to running pkgs/development/node-packages/generate.sh.
• I saw a link to the NixOS/cabal2nix README somewhere, so I looked there, but it didn't mention the regenerate-nixpkgs.sh script either
• Even when I found the regenerate-nixpkgs.sh, I spent a few minutes trying to find out how to install cabal, because haskellPackages.cabal is broken.

=> These things would have improved the experience by a lot:

• Somehow including or linking the regenerate-nixpkgs.sh script in pkgs/development/haskell-modules/
• Making it clone cabal2nix master in a temp dir
• A nix-shell shebang with cabal-install

Hey everyone! @cdepillabout and @sternenseemann. I had a fresh look at this and I concluded, that this PR does exactly what we want. Now with a break in the workflow I recommend we merge it asap and switch to updating the hackage-package-set using these scripts.

What @hyperfekt has done here seems exactly right. Thank you for your work! Since you have probably long given up on this I will take over and drive this home, if you don‘t mind.

I think having a cabal2nix-latest cached by hydra is advantageous.

If you don‘t object I will clean up the merge conflicts and test this out. (I want to make sure this doesn‘t break callCabal2nix, which it could since it’s touching the hashes.)

The only thing we could bikeshed about are filenames and locations. I kinda hope that we will add a few more scripts in the future so I am a bit weary of having them all plain maintainers/scripts

One thing to note: We probably also want to migrate the update-stackage script into nixpkgs.

Subsumed by #121391.