Skip to content

[staging-20.09] Add ghc8102BinaryMinimal to stay within hydra limits on aarch64 #102504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 5, 2020
Merged

[staging-20.09] Add ghc8102BinaryMinimal to stay within hydra limits on aarch64 #102504

merged 2 commits into from
Nov 5, 2020
+9 −14

Conversation

roberth
Copy link
Member

@roberth roberth commented Nov 2, 2020 •
edited
Loading

Motivation for this change

Make ghc buildable on hydra for aarch64.

It's currently failing with Output limit exceeded https://hydra.nixos.org/build/129487354#tabs-summary
Hydra only allows outputs up to 231 NAR bytes.
The new ghc8102BinaryMinimal is 2136668000 out of 2147483648 bytes; ~10M under the limit.
This is achieved by removing the documentation and the profiling objects, neither of which are used for bootstrapping.

I have a build of GHC 8.8.4 in that's in progress.

TODO

  • finish compilation of ghc884
  • inline docs
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Sorry, something went wrong.

@roberth roberth changed the title Add ghc8102BinaryMinimal to stay within hydra limits [20.09] Add ghc8102BinaryMinimal to stay within hydra limits on aarch64 Nov 2, 2020
@roberth roberth mentioned this pull request Nov 2, 2020
Merged
10 tasks
@SuperSandro2000
Copy link
Member

I wanted to build this locally but it does not eval. Tried it on x86_64-linux

@roberth
Copy link
Member Author

roberth commented Nov 2, 2020

@SuperSandro2000 fixed in haskell.compiler.integer-simple: Exclude ghc8102Binary*

@roberth roberth force-pushed the minimal-ghc-to-please-hydra-limits branch from 6b80eb8 to 95b230c Compare November 2, 2020 22:09
@cole-h cole-h mentioned this pull request Nov 2, 2020
Merged
10 tasks
@cole-h
Copy link
Member

cole-h commented Nov 2, 2020

In order to fix release-20.09 evals, I opened a separate PR with 95b230c in #102539; you might want to rebase to drop that commit ;)

(Thanks for the fix!)

@roberth roberth force-pushed the minimal-ghc-to-please-hydra-limits branch from 95b230c to a945a85 Compare November 3, 2020 09:41
@roberth
Copy link
Member Author

roberth commented Nov 3, 2020

I've

  • tested this PR locally and it can still bootstrap ghc despite removing the files that take it below the limit 🎉
  • rebased it, cleaned up and has inline docs
  • hydraPlatforms = [] when the docs are stripped, for legal reasons

Presumably hydra will still allow it to be built for use as a dependency.

Thanks @cole-h. I suppose ofborg was failing in the comparison step because the old version was bad, so it seems like ofborg should work now 🎉

@roberth roberth marked this pull request as ready for review November 3, 2020 09:50
@roberth roberth requested a review from cdepillabout as a code owner November 3, 2020 09:50
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Nov 3, 2020
@vcunat
Copy link
Member

vcunat commented Nov 3, 2020

I suspect it will get into cache.nixos.org anyway but we'll see.

@roberth
Copy link
Member Author

roberth commented Nov 3, 2020

I suspect it will get into cache.nixos.org anyway but we'll see.

If hydra only pushes closures of attributes with non-nil hydraPlatforms, it should work out, because it's not a runtime dependency.
We should enforce that with disallowedRequisites. I'll check if that can work.

@roberth
Copy link
Member Author

roberth commented Nov 3, 2020 •
edited
Loading

disallowedRequisites causes a mass rebuild, so I'll skip it for 20.09, but it should be added to the forward port, to master. 20.09 should only receive backports from master anyway, catching any problems.

@vcunat
Copy link
Member

vcunat commented Nov 3, 2020

We could switch to fixing staging-next before 20.09, as it apparently got unblocked.

@roberth
Copy link
Member Author

roberth commented Nov 3, 2020

I'm happy to add the forward port there, but I don't think we should delay this PR for it.

@vcunat
Copy link
Member

vcunat commented Nov 3, 2020 

$ nix build -f https://github.com/hercules-ci/nixpkgs/archive/minimal-ghc-to-please-hydra-limits.tar.gz ghc
error: --- ThrownError ------------------------------------------------------------------------------------ nix
Package ‘ghc-8.8.4’ in /nix/store/70bs1f100yldwvihsp03lpgsfsklps2b-source/pkgs/development/compilers/ghc/8.8.4.nix:251 is not supported on ‘aarch64-linux’, refusing to evaluate.

@vcunat
Copy link
Member

vcunat commented Nov 3, 2020

Why did you touch platforms instead of hydraPlatforms?

@roberth roberth force-pushed the minimal-ghc-to-please-hydra-limits branch from a945a85 to 22e9260 Compare November 3, 2020 10:51
@roberth
Copy link
Member Author

roberth commented Nov 3, 2020

Why did you touch platforms instead of hydraPlatforms?

A mistake I didn't catch because I evaluated on x86_64-linux. Fixed.

@roberth roberth mentioned this pull request Nov 3, 2020
Merged
10 tasks
@roberth
Copy link
Member Author

roberth commented Nov 3, 2020

@vcunat I'm building this locally and I've created #102614 for staging-next.

@vcunat
Copy link
Member

vcunat commented Nov 3, 2020

Uh, I got some glitch

builder for '/nix/store/yzzbli3jy90w2gfiskpqwl34v79dxy18-jailbreak-cabal-1.3.5.drv' failed
  [1 of 1] Compiling Main             ( Setup.hs, /build/Main.o )
  Linking Setup ...
  
  out of memory allocating 1605596214263808 bytes after a total of 18446462599109164456 bytes
  collect2: error: ld returned 1 exit status

but it disappeared on retrying, so I expect I'll ignore it.

@roberth
Copy link
Member Author

roberth commented Nov 3, 2020 •
edited
Loading

It built successfully on my machine.

@roberth
Copy link
Member Author

roberth commented Nov 3, 2020 •
edited
Loading

I'll open a PR for staging-20.09 I've pushed to staging-20.09 because this PR rebuilds haskell on other systems because of the disallowedRequisites drv attr

@roberth
Copy link
Member Author

roberth commented Nov 3, 2020 •
edited
Loading

Hydra (not this PRs base branch though!): https://hydra.nixos.org/eval/1624234

@vcunat
Copy link
Member

vcunat commented Nov 4, 2020 •
edited
Loading

I suspect it will get into cache.nixos.org anyway but we'll see.

$ nix build /nix/store/5c5bfc6n1ynr2anajix18l6brdv6dqhq-ghc-8.10.2-binary
[... fetched from cache.nixos.org]

(this one was used for current nixpkgs master)

@vcunat
Copy link
Member

vcunat commented Nov 4, 2020

I think the problem here is that we utilize cache.nixos.org to distribute all build dependencies to the builders, so we do need to upload all the build products.

@roberth
ghc8102BinaryMinimal: Re-add docs, remove more profiling files
7bd21d3
@roberth
ghc8102Binary*: simplify meta
4643796 
Not distributing dependencies via hydraPlatforms does not work,
so we keep meta simple.
@roberth
Copy link
Member Author

roberth commented Nov 4, 2020

Some good news. I had overlooked a class of profiling files, so I've added the docs back in and we're at only 74% of the limit now.
My GHC build has been going for a while now. So far so good.

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 and removed 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Nov 4, 2020
@roberth roberth changed the base branch from release-20.09 to staging-20.09 November 4, 2020 22:14
@roberth roberth changed the title [20.09] Add ghc8102BinaryMinimal to stay within hydra limits on aarch64 [staging-20.09] Add ghc8102BinaryMinimal to stay within hydra limits on aarch64 Nov 4, 2020
@roberth roberth mentioned this pull request Nov 4, 2020
Merged
10 tasks
@roberth
Copy link
Member Author

roberth commented Nov 5, 2020

✔️ tested

@roberth roberth merged commit 055f5f9 into NixOS:staging-20.09 Nov 5, 2020
@roberth roberth mentioned this pull request Nov 5, 2020
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cdepillabout cdepillabout

Assignees
No one assigned
Labels
6.topic: haskell 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@roberth @SuperSandro2000 @cole-h @vcunat