Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

haskell: tune generic-builder for more parallel build performance #86948

Merged
merged 1 commit into from May 7, 2020
Merged

haskell: tune generic-builder for more parallel build performance #86948

merged 1 commit into from May 7, 2020

Conversation

peti
Copy link
Member

@peti peti commented May 5, 2020 •
edited

Raise the maximum number of cores used for parallel building from 4 to 16. Increase the size of the allocation area for GHC's garbage collector from 1 MiB to 64 MiB.

See https://www.twitch.tv/videos/611899011 for the motivation for this change.

@peti peti requested review from cdepillabout and nh2 May 5, 2020 14:33
nh2
nh2 reviewed May 5, 2020
View reviewed changes
Copy link
Contributor

@nh2 nh2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks sensible to me, but one question:

If it is done at the ghcWithPackages level, do/can/will we also have the ability to override it per Haskell package? I am suspecting that there might be certain large packages for which we may need to tune down -j or -A to not consume too much memory on builders that have less than the casual 3 TB RAM.

@peti peti mentioned this pull request May 5, 2020
Merged
3 tasks
@peti
Copy link
Member Author

peti commented May 7, 2020

It just occurred to me that the change to the ghcWithPackages wrapper is unnecessary, because you can do the same thing in ~/.cabal/config. Adding

program-default-options
  ghc-options: -j16 +RTS -A64M -RTS

there has the same effect -- at least if you're building with cabal-install. I'm pretty sure an identical mechanism exists for stack, too. So, on second thought I'd rather not introduce yet another location to configure default flags that's specific to Nixpkgs.

@peti peti changed the title ghcWithPackages: add ghcOptions argument to pass extra options to the compiler haskell: tune generic-builder for more parallel build performance May 7, 2020
@peti peti requested review from cdepillabout and nh2 May 7, 2020 11:21
@peti
haskell: tune generic-builder for more parallel build performance
f6df169 
Raise the maximum number of cores used for parallel building from 4 to 16.
Increase the size of the allocation area for GHC's garbage collector from 1 MiB
to 64 MiB.

See https://www.twitch.tv/videos/611899011 for the motivation for this change.
@peti
Copy link
Member Author

peti commented May 7, 2020

I'll merge this change to haskell-updates so that we can get some test results from Hydra before the live stream tomorrow.

@peti peti merged commit 6a815f9 into NixOS:haskell-updates May 7, 2020
@peti peti deleted the t/parallel-ghc-builds branch May 7, 2020 17:40
@expipiplus1
Copy link
Contributor

This change causes an uncomfortably high load on my poor machine.

image

I've also noticed loads of 120+ on my 8(16) core desktop, surely this can't be faster.

(sorry if I posted this before somewhere, I thought I did for another machine but couldn't find anything)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cdepillabout cdepillabout cdepillabout approved these changes

@nh2 nh2 Awaiting requested review from nh2

Assignees
No one assigned
Labels
6.topic: haskell 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@peti @expipiplus1 @cdepillabout @nh2