Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport haskell backpack and profiling fixes #46163

Merged
merged 4 commits into from Sep 10, 2018

Conversation

shlevy
Copy link
Member

@shlevy shlevy commented Sep 6, 2018

Alex Biehl and others added 4 commits September 6, 2018 11:40
`all-functions` corresponds to `-fprof-auto` which places an SCC on every binding. It is well known that SCCs hinder GHC from doing its optimization magic and really slows down profiled code to a point where the profiling reports are completely skewed towards things that were completely optimized away in production settings. Concretely this shows up with things like lenses which do not carry runtime overhead when properly simplified.

`exported-functions` corresponds to GHCs `-fprof-auto-exported` which doesn't put SCCs on `INLINE`d code and in turn doesn't influence simplification of this basic but important stuff.

(cherry picked from commit 3c70c4c)
@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: ghc, haskell.compiler.ghc861

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


@shlevy
Copy link
Member Author

shlevy commented Sep 6, 2018

@samueldr @vcunat Please advise on timing for merging this.

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-linux (full log)

Attempted: ghc, haskell.compiler.ghc861

Partial log (click to expand)

<<ghc: 1380788832 bytes, 115 GCs, 20721034/69141960 avg/max bytes residency (8 samples), 147M in use, 0.000 INIT (0.001 elapsed), 2.952 MUT (3.230 elapsed), 1.231 GC (1.312 elapsed) :ghc>>
"/nix/store/kd7w21i5namb6c18cj05cc0708rwz291-ghc-8.2.1-binary/bin/ghc" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall  -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -package-db libraries/bootstrapping.conf  -this-unit-id ghc-8.2.2 -hide-all-packages -i -icompiler/backpack -icompiler/basicTypes -icompiler/cmm -icompiler/codeGen -icompiler/coreSyn -icompiler/deSugar -icompiler/ghci -icompiler/hsSyn -icompiler/iface -icompiler/llvmGen -icompiler/main -icompiler/nativeGen -icompiler/parser -icompiler/prelude -icompiler/profiling -icompiler/rename -icompiler/simplCore -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn -icompiler/stranal -icompiler/typecheck -icompiler/types -icompiler/utils -icompiler/vectorise -icompiler/stage1/build -Icompiler/stage1/build -icompiler/stage1/build/./autogen -Icompiler/stage1/build/./autogen -Icompiler/. -Icompiler/parser -Icompiler/utils -Icompiler/stage1    -optP-include -optPcompiler/stage1/build/./autogen/cabal_macros.h -package-id base-4.10.0.0 -package-id deepseq-1.4.3.0 -package-id directory-1.3.0.2 -package-id process-1.6.1.0 -package-id bytestring-0.10.8.2 -package-id binary-0.8.5.1 -package-id time-1.8.0.2 -package-id containers-0.5.10.2 -package-id array-0.5.2.0 -package-id filepath-1.4.1.2 -package-id template-haskell-2.12.0.0 -package-id hpc-0.6.0.3 -package-id transformers-0.5.2.0 -package-id ghc-boot-8.2.2 -package-id ghc-boot-th-8.2.2 -package-id ghci-8.2.2 -package-id hoopl-3.10.2.2 -package-id unix-2.7.2.2 -package-id terminfo-0.4.1.0 -Wall -fno-warn-name-shadowing -this-unit-id ghc -XHaskell2010 -DSTAGE=1 -Rghc-timing   -no-user-package-db -rtsopts       -odir compiler/stage1/build -hidir compiler/stage1/build -stubdir compiler/stage1/build    -c compiler/specialise/SpecConstr.hs -o compiler/stage1/build/SpecConstr.o
building of '/nix/store/daw2rir4042bqcxvw0dqv98i8ff5bcil-ghc-8.2.2.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/5kmphd7km5vcb6q96rx65djfpfjbib58-hscolour-1.24.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/p3caxz831p2yh2ql5407fz3a5jrivw07-hscolour-1.24.4.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/0da1mkg8rd5nv2092dfh2m5ay0b50bn8-mtl-2.2.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/qs7ly6rgl0p8f3ghaz822ykl416d1gnr-happy-1.19.9.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/3csp74gv9l8sxzibc1fkwzwlj1qrnij6-alex-3.2.4.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/kgdggx7zaah123sslhpdp5n6d18ifwgm-ghc-8.6.0.20180810.drv': 4 dependencies couldn't be built
error: build of '/nix/store/802wif5k4qqqwhbdgpz5lmxc56lby9in-ghc-8.4.3.drv', '/nix/store/kgdggx7zaah123sslhpdp5n6d18ifwgm-ghc-8.6.0.20180810.drv' failed

@GrahamcOfBorg
Copy link

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: ghc, haskell.compiler.ghc861

Partial log (click to expand)

"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id transformers-0.5.5.0 -hide-all-packages -i -ilibraries/transformers/. -ilibraries/transformers/dist-install/build -Ilibraries/transformers/dist-install/build -ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/.    -optP-include -optPlibraries/transformers/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -XHaskell98 -O2  -no-user-package-db -rtsopts  -Wno-unused-matches -Wno-unused-imports -Wno-redundant-constraints -Wno-orphans -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/transformers/dist-install/build -hidir libraries/transformers/dist-install/build -stubdir libraries/transformers/dist-install/build   -dynamic-too -c libraries/transformers/./Control/Monad/Trans/RWS.hs -o libraries/transformers/dist-install/build/Control/Monad/Trans/RWS.o -dyno libraries/transformers/dist-install/build/Control/Monad/Trans/RWS.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id transformers-0.5.5.0 -hide-all-packages -i -ilibraries/transformers/. -ilibraries/transformers/dist-install/build -Ilibraries/transformers/dist-install/build -ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/.    -optP-include -optPlibraries/transformers/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -XHaskell98 -O2  -no-user-package-db -rtsopts  -Wno-unused-matches -Wno-unused-imports -Wno-redundant-constraints -Wno-orphans -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/transformers/dist-install/build -hidir libraries/transformers/dist-install/build -stubdir libraries/transformers/dist-install/build   -dynamic-too -c libraries/transformers/./Control/Monad/Trans/List.hs -o libraries/transformers/dist-install/build/Control/Monad/Trans/List.o -dyno libraries/transformers/dist-install/build/Control/Monad/Trans/List.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id transformers-0.5.5.0 -hide-all-packages -i -ilibraries/transformers/. -ilibraries/transformers/dist-install/build -Ilibraries/transformers/dist-install/build -ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/.    -optP-include -optPlibraries/transformers/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -XHaskell98 -O2  -no-user-package-db -rtsopts  -Wno-unused-matches -Wno-unused-imports -Wno-redundant-constraints -Wno-orphans -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/transformers/dist-install/build -hidir libraries/transformers/dist-install/build -stubdir libraries/transformers/dist-install/build   -dynamic-too -c libraries/transformers/./Control/Monad/Trans/Error.hs -o libraries/transformers/dist-install/build/Control/Monad/Trans/Error.o -dyno libraries/transformers/dist-install/build/Control/Monad/Trans/Error.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id transformers-0.5.5.0 -hide-all-packages -i -ilibraries/transformers/. -ilibraries/transformers/dist-install/build -Ilibraries/transformers/dist-install/build -ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/dist-install/build/./autogen -Ilibraries/transformers/.    -optP-include -optPlibraries/transformers/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -XHaskell98 -O2  -no-user-package-db -rtsopts  -Wno-unused-matches -Wno-unused-imports -Wno-redundant-constraints -Wno-orphans -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/transformers/dist-install/build -hidir libraries/transformers/dist-install/build -stubdir libraries/transformers/dist-install/build   -dynamic-too -c libraries/transformers/./Control/Monad/Trans/Identity.hs -o libraries/transformers/dist-install/build/Control/Monad/Trans/Identity.o -dyno libraries/transformers/dist-install/build/Control/Monad/Trans/Identity.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id haskeline-0.7.4.2 -hide-all-packages -i -ilibraries/haskeline/. -ilibraries/haskeline/dist-install/build -Ilibraries/haskeline/dist-install/build -ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/includes -Ilibraries/haskeline/dist-install/build/includes   -optP-DTERMINFO -optP-include -optPlibraries/haskeline/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -package-id containers-0.5.11.0 -package-id directory-1.3.1.5 -package-id bytestring-0.10.8.2 -package-id filepath-1.4.2 -package-id transformers-0.5.5.0 -package-id process-1.6.3.0 -package-id stm-2.4.5.0 -package-id unix-2.7.2.2 -package-id terminfo-0.4.1.1 -Wall -XHaskell98 -XForeignFunctionInterface -XRank2Types -XFlexibleInstances -XTypeSynonymInstances -XFlexibleContexts -XExistentialQuantification -XScopedTypeVariables -XGeneralizedNewtypeDeriving -XStandaloneDeriving -XMultiParamTypeClasses -XOverlappingInstances -XUndecidableInstances -XCPP -XDeriveDataTypeable -XPatternGuards -O2  -no-user-package-db -rtsopts  -Wno-deprecations -Wno-unused-imports -Wno-redundant-constraints -Wno-simplifiable-class-constraints -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/haskeline/dist-install/build -hidir libraries/haskeline/dist-install/build -stubdir libraries/haskeline/dist-install/build   -dynamic-too -c libraries/haskeline/./System/Console/Haskeline/MonadException.hs -o libraries/haskeline/dist-install/build/System/Console/Haskeline/MonadException.o -dyno libraries/haskeline/dist-install/build/System/Console/Haskeline/MonadException.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id haskeline-0.7.4.2 -hide-all-packages -i -ilibraries/haskeline/. -ilibraries/haskeline/dist-install/build -Ilibraries/haskeline/dist-install/build -ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/includes -Ilibraries/haskeline/dist-install/build/includes   -optP-DTERMINFO -optP-include -optPlibraries/haskeline/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -package-id containers-0.5.11.0 -package-id directory-1.3.1.5 -package-id bytestring-0.10.8.2 -package-id filepath-1.4.2 -package-id transformers-0.5.5.0 -package-id process-1.6.3.0 -package-id stm-2.4.5.0 -package-id unix-2.7.2.2 -package-id terminfo-0.4.1.1 -Wall -XHaskell98 -XForeignFunctionInterface -XRank2Types -XFlexibleInstances -XTypeSynonymInstances -XFlexibleContexts -XExistentialQuantification -XScopedTypeVariables -XGeneralizedNewtypeDeriving -XStandaloneDeriving -XMultiParamTypeClasses -XOverlappingInstances -XUndecidableInstances -XCPP -XDeriveDataTypeable -XPatternGuards -O2  -no-user-package-db -rtsopts  -Wno-deprecations -Wno-unused-imports -Wno-redundant-constraints -Wno-simplifiable-class-constraints -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/haskeline/dist-install/build -hidir libraries/haskeline/dist-install/build -stubdir libraries/haskeline/dist-install/build   -dynamic-too -c libraries/haskeline/./System/Console/Haskeline/Prefs.hs -o libraries/haskeline/dist-install/build/System/Console/Haskeline/Prefs.o -dyno libraries/haskeline/dist-install/build/System/Console/Haskeline/Prefs.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id haskeline-0.7.4.2 -hide-all-packages -i -ilibraries/haskeline/. -ilibraries/haskeline/dist-install/build -Ilibraries/haskeline/dist-install/build -ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/includes -Ilibraries/haskeline/dist-install/build/includes   -optP-DTERMINFO -optP-include -optPlibraries/haskeline/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -package-id containers-0.5.11.0 -package-id directory-1.3.1.5 -package-id bytestring-0.10.8.2 -package-id filepath-1.4.2 -package-id transformers-0.5.5.0 -package-id process-1.6.3.0 -package-id stm-2.4.5.0 -package-id unix-2.7.2.2 -package-id terminfo-0.4.1.1 -Wall -XHaskell98 -XForeignFunctionInterface -XRank2Types -XFlexibleInstances -XTypeSynonymInstances -XFlexibleContexts -XExistentialQuantification -XScopedTypeVariables -XGeneralizedNewtypeDeriving -XStandaloneDeriving -XMultiParamTypeClasses -XOverlappingInstances -XUndecidableInstances -XCPP -XDeriveDataTypeable -XPatternGuards -O2  -no-user-package-db -rtsopts  -Wno-deprecations -Wno-unused-imports -Wno-redundant-constraints -Wno-simplifiable-class-constraints -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/haskeline/dist-install/build -hidir libraries/haskeline/dist-install/build -stubdir libraries/haskeline/dist-install/build   -dynamic-too -c libraries/haskeline/./System/Console/Haskeline/Monads.hs -o libraries/haskeline/dist-install/build/System/Console/Haskeline/Monads.o -dyno libraries/haskeline/dist-install/build/System/Console/Haskeline/Monads.dyn_o
"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall      -this-unit-id haskeline-0.7.4.2 -hide-all-packages -i -ilibraries/haskeline/. -ilibraries/haskeline/dist-install/build -Ilibraries/haskeline/dist-install/build -ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/dist-install/build/./autogen -Ilibraries/haskeline/includes -Ilibraries/haskeline/dist-install/build/includes   -optP-DTERMINFO -optP-include -optPlibraries/haskeline/dist-install/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -package-id containers-0.5.11.0 -package-id directory-1.3.1.5 -package-id bytestring-0.10.8.2 -package-id filepath-1.4.2 -package-id transformers-0.5.5.0 -package-id process-1.6.3.0 -package-id stm-2.4.5.0 -package-id unix-2.7.2.2 -package-id terminfo-0.4.1.1 -Wall -XHaskell98 -XForeignFunctionInterface -XRank2Types -XFlexibleInstances -XTypeSynonymInstances -XFlexibleContexts -XExistentialQuantification -XScopedTypeVariables -XGeneralizedNewtypeDeriving -XStandaloneDeriving -XMultiParamTypeClasses -XOverlappingInstances -XUndecidableInstances -XCPP -XDeriveDataTypeable -XPatternGuards -O2  -no-user-package-db -rtsopts  -Wno-deprecations -Wno-unused-imports -Wno-redundant-constraints -Wno-simplifiable-class-constraints -Wno-deprecated-flags     -Wnoncanonical-monad-instances  -odir libraries/haskeline/dist-install/build -hidir libraries/haskeline/dist-install/build -stubdir libraries/haskeline/dist-install/build   -dynamic-too -c libraries/haskeline/./System/Console/Haskeline/LineState.hs -o libraries/haskeline/dist-install/build/System/Console/Haskeline/LineState.o -dyno libraries/haskeline/dist-install/build/System/Console/Haskeline/LineState.dyn_o
building of '/nix/store/cjmx6qn2gd49y9rd9bk7ff0bj5rbs9k3-ghc-8.4.3.drv' timed out after 3600 seconds
error: build of '/nix/store/cjmx6qn2gd49y9rd9bk7ff0bj5rbs9k3-ghc-8.4.3.drv', '/nix/store/kd1dv2ncdsq3gkvwqdd2csvns7amd78j-ghc-8.6.0.20180810.drv' failed

@samueldr
Copy link
Member

samueldr commented Sep 8, 2018

I'm not familiar with mass rebuilds (if this is one), and the usual timings and such. Any tip?

@shlevy
Copy link
Member Author

shlevy commented Sep 8, 2018

This is a mass-haskell rebuild. I created the relevant hydra jobset already and the cache is mostly populated.

There's been some discussion around this lately so it might have changed, but in the past we've just merged major rebuilds into staging (or, for haskell, haskell-updates), then when the relevant jobset stabilized and binaries uploaded anyone could just merge it into master or the relevant release branch. Since we're still in release prep mode I didn't want to just merge in churn unwarned, though.

@shlevy shlevy merged commit 7f34562 into release-18.09 Sep 10, 2018
@vcunat vcunat deleted the haskell-updates-18.09 branch September 10, 2018 12:58
@vcunat
Copy link
Member

vcunat commented Sep 10, 2018

Ah, yes, timing seems good. I've been falling behind notification updates.

There's this tarball problem blocking channels on both master and 18.09 anyway...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants