Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Haskell package set to Stackage Nightly (plus other fixes) #103740

Merged
merged 43 commits into from Nov 20, 2020

Conversation

peti
Copy link
Member

@peti peti commented Nov 13, 2020

This PR is test-built by Hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. I'll fix up the remaining errors and merge it on Friday, 2020-11-20 20:00 +01:00. You can watch this live on Twitch at https://www.twitch.tv/peti343. In addition to the chat features offered by Twitch, there is also a voice conference at https://discord.gg/YTEa3XR that viewers can use to chat with me and with each other.

@cdepillabout
Copy link
Member

@peti Why did you decide to update the package set to Nightly (instead of an LTS)?

Also, when LTS-17 comes out, are you planning on switching back to track it?

@peti
Copy link
Member Author

peti commented Nov 14, 2020

The discussion took place at http://www.twitch.tv/peti343/v/801715350?sr=a&t=3625s.

We felt that Nightly is better for us in general. Release branches could follow some LTS set, of course, but we actually don't update the package set in release branches, so I guess it doesn't make any practical difference.

@cdepillabout
Copy link
Member

cdepillabout commented Nov 14, 2020

@peti Thanks for the link to the conversation.

I think I agree with most of the points brought up. It sounds like Nightly might actually be a better fit for nixpkgs, since we also provide many packages from Hackage that are not in Stackage. Packages from Hackage are sometimes hard to get working with the slightly older stuff from Stackage.

One concern I have is how often Nightly drops "important" packages. I remember from about a year or so ago I was quite surprised when Nightly dropped Servant, which has quite a few dependencies in Stackage. However, I don't actually have any idea how frequent or widespread this kind of breakage is. Servant is somewhat of a fringe package (even though it has a lot of dependencies), but I wonder what the story is for the more "core" packages, like lens, aeson, conduit, etc.

On Twitch @maralorn suggested that we should do another call for help with making sure our various Haskell packages compile on Nightly / ghc-8.10.2. I think this sounds like a good idea.

@maralorn
Copy link
Member

Okay, I got this down to zero eval errors.

We are now at 5933 building packages, down from 6044 before the switch to stackage nightly. This is not so bad.

Sadly right now some of the most important packages are still broken. e.g. cabal-install, stack, hls, git-annex, neuron …

So, still a lot to do.

@maralorn
Copy link
Member

maralorn commented Nov 17, 2020

I will just very informally document here, what packages are still on my list to be fixed:

  • cabal-install
  • hls
  • stack
  • neuron
  • stan
  • git-annex
  • password
  • snap

Of course, I don‘t know if I can fix them all.

@cdepillabout
Copy link
Member

@srid

neuron isn't building because the following packages are broken on this branch:

  • reflex
  • aeson-gadt-th
  • rib-core
  • dependent-sum-aeson-orphans

@cdepillabout
Copy link
Member

I've fixed Spago in #104164.

@maralorn
Copy link
Member

@srid

neuron isn't building because the following packages are broken on this branch:

* `reflex`

* `aeson-gadt-th`

* `rib-core`

* `dependent-sum-aeson-orphans`

I have fixed all of them locally. But most of them with simple dontCheck and doJailbreak, so I want to clean that up and make a pretty PR with patches or at least issues out of it.

@maralorn
Copy link
Member

I have fixed all neuron deps other than neuron itself. It fails with this error message: srid/neuron#475

@cdepillabout
Copy link
Member

@maralorn The fix in 80d6b9a for pcg-random was all that was needed to get password building. I'll check it off on your list above.

@expipiplus1
Copy link
Contributor

expipiplus1 commented Nov 19, 2020

streamly-0.7.2 doesn't compile with primitive-0.7.1 (even after jailbreak) (hence pkgs.nix-linter is broken)

PR here #104232

@maralorn
Copy link
Member

As soon as we merge master into haskell-updates (or the other way around). hls will break. :-(

#104275

@maralorn
Copy link
Member

That has been fixed. \o/

I think it's a bit late now to merge master in, if we did that now, we won‘t have reasonable hydra infos tomorrow.

I wish we had a way for hydra to tell us which jobs fail now which built successfully last week. So we could have a look at which of them should also be fixed …

@maralorn
Copy link
Member

@peti You obviously didn‘t read what I said or decided to disagree. 😆

@peti
Copy link
Member Author

peti commented Nov 20, 2020

I don't see an alternative. We have to merge master in, no? What else could we do?

peti and others added 15 commits November 20, 2020 20:57
This update was generated by hackage2nix v2.15.5-27-g11a103b from Hackage revision
commercialhaskell/all-cabal-hashes@e037d20.
The build was broken because our `primitive` version was too high.

Using the patch from composewell/streamly#565
This bumps aeson-gadt-th, which will be pulled in by the hackage update
this night anyways.
This update was generated by hackage2nix v2.15.5-27-g11a103b from Hackage revision
commercialhaskell/all-cabal-hashes@fa72061.
This update was generated by hackage2nix v2.15.5-28-gd194902 from Hackage revision
commercialhaskell/all-cabal-hashes@8c0e05f.
@Kiwi, please check the state of glirc. matterhorn, mattermost-api, and mattermost-api-qc.

@roberth, please check hercules-ci-agent, hercules-ci-api-core, and hercules-ci-api-agent.
This update was generated by hackage2nix v2.15.5-28-gd194902 from Hackage revision
commercialhaskell/all-cabal-hashes@8c0e05f.
The cachix build is broken with ghc-8.10.x and recent versions of
its dependencies, particularly servant, swagger2, etc. We cannot
easily fix that with overrides -- it would be best if upstream
released a new version that builds successfully in the Stackage
Nightly environment. A simple way to ensure that would be to add
cachix to Stackage.

Ping @domenkozar.
@domenkozar
Copy link
Member

I've only gotten the notification that decision was made to switch to nightly. Can I get like a week before this is merged? Otherwise cachix will be broken and I literally had no time to prepare without knowing about the decision.

@turboMaCk
Copy link
Member

turboMaCk commented Nov 20, 2020

how about this quickfix:

  # in all-packages.nix
  cachix = haskell.lib.justStaticExecutables haskell.packages.ghc883.cachix;

this builds for me and might be good enough perhaps?

@peti
Copy link
Member Author

peti commented Nov 20, 2020

@domenkozar, there are plenty of ways to fix cachix quickly. One has been pointed out by @turboMaCk. Another option might be to use overrideScope to re-create the environment (set of versions of its dependencies) in which it builds successfully. I don't know. It doesn't feel like a big deal if cachix is broken for a few hours or even days on the unstable development branch. I'd rather not abort all our efforts here to avoid a problem that doesn't feel particularly major to be honest.

@peti peti merged commit 4f2880b into master Nov 20, 2020
@domenkozar
Copy link
Member

@peti you're a true gentlement :D

@maralorn
Copy link
Member

maralorn commented Nov 20, 2020

For what it's worth. It is really annoying that we don‘t have notifications for failed hydra builds. We should have an automated way to notify devs about pending broken merges as early as possible.

Our current workflow is lacking in that regard and we will need to improve on that.

@peti
Copy link
Member Author

peti commented Nov 20, 2020

We'll merge #104424 next week.

@domenkozar
Copy link
Member

@maralorn we have release blockers, but it doesn't help if someone removes your package from it 20min before merging the PR...

@domenkozar
Copy link
Member

And disabling servant means disabling A LOT of packages.

@expipiplus1
Copy link
Contributor

For what it's worth. It is really annoying that we don‘t have notifications for failed hydra builds. We should have an automated way to notify devs about pending broken merges as early as possible.

Our current workflow is lacking in that regard and we will need to improve on that.

Can you open an issue for this? (It's also been discussed elsewhere, for instance the static pkgs maintainer issue)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-ghc-default-version-8-10-2-package-versions-now-from-stackage-nightly/10117/4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants