[wip] haskell: add core libraries logic to remove need for global package db #43300

matthewbauer · 2018-07-10T19:16:01Z

This moves core libraries in Haskell to its own derivations. This is
needed to avoid references to the GHC compiler ending up in binaries
built by GHC. Instead of referencing GHC directly, these binaries will
now link to the copies.

Unfortunately, for this to work, we need to know which GHC libraries
exist at evaluation time. This requires maintaining a list of the
builtin library names & their versions in the GHC compiler’s “libs”
attribute. A better way of handling this is needed.

This is still a work in progress.

/cc @kirelagin @bgamari @ElvishJerricco

peti · 2018-07-10T19:34:32Z

How do these changes relate to #43070?

matthewbauer · 2018-07-10T19:42:18Z

I think they accomplish the same thing - just in different ways. Both should reduce closure sizes.

Pros of this approach:

Avoids rebuilding GHC.
Meant as a stopgap fix until we can split the GHC compiler from its core libraries (more work on GHC is needed for this to be possible IIUC).
Doesn't rely on multiple outputs (perhaps this is a con for some but IMO having a separate output for each library is kind of an abuse of the idea of multiple outputs).

Cons of this approach:

You end up with two copies of each core library in your Nix store.
Requires keeping track of name + version of every core library in the GHC compiler (more maintenance especially because it looks like they change from version to version).

matthewbauer · 2018-07-10T20:57:59Z

This is currently blocked on an issue that I have been unable to debug:

Configuring hscolour-1.24.4...
Dependency base <10: using base-4.12.0.0
Dependency containers -any: using containers-0.6.0.1
Dependency base <10: using base-4.12.0.0
Dependency containers -any: using containers-0.6.0.1
<command line>: unknown package: rts

It looks like it picks up base but not rts?

Ericson2314 · 2018-07-11T15:07:46Z

@peti Unless we're worried about churn for the builders, I'd love to land #43070 as soon as possible, as it was passing deps before mere conflicts. Then we can do this approach, with the libraries having been listed for us, right after.

With NixOS/cabal2nix#358 enabling a new strictDeps PR with far fewer overrides + all libraries coming from separate non-GHC derivations, we should be able to both 1) finish cross support and 2) remove all duplicate args, in less than a week! [I say finish because while it would be nice to e.g. build components, including Setup, separately, there is no burden for correctness forcing us to do so.]

domenkozar · 2018-07-11T15:22:48Z

This is really good work, it will reduce haskell closures significantly for packages dynamically linked.

kirelagin · 2018-07-11T15:32:48Z

Both should reduce closure sizes.

Actually, I don’t think that my approach affects closure size in any way.

kirelagin · 2018-07-11T15:35:56Z

I might be missing something, but can’t we just “merge” our two PRs by moving the code from mkCoreLib here into the install phase in my PR? This way boot libraries outputs will not retain a link to the main output and we will get the closure size reduced.

P. S. I wonder if the GHC build system can be told to install those packages straight into the right location instead of us copying them around manually.

kirelagin · 2018-07-11T15:41:42Z

having a separate output for each library is kind of an abuse of the idea of multiple outputs

I don’t see how this is an abuse TBH. The GHC build process is structured in such a way that it produces both the compiler and libraries for it to use and, furthermore, it is the only supported way of getting those libraries. What I mean is that some of the boot libraries can be then recompiled, but some of them just cannot be built outside of the ghc compilation process, and, AFAIU, this is exactly the purpose of multiple outputs.

What we can do is limit those libraries that go to multiple outputs strictly to those that cannot be built separately.

Ericson2314 · 2018-07-11T16:11:40Z

@kirelagin I personally don't mind multiple outputs, but I do think it's important that we have it so generic-builder and ghc with generic-builder can find them exactly one way. Whether we do that with multiple outputs, or shim derivations I guess doesn't matter as much on that front as long as GHC-without-libraries has its own output / shim derivation. Put another way, the -B for GHC should give it a thing without libraries.

matthewbauer · 2018-07-11T16:19:53Z

I don’t see how this is an abuse TBH. The GHC build process is structured in such a way that it produces both the compiler and libraries for it to use and, furthermore, it is the only supported way of getting those libraries. What I mean is that some of the boot libraries can be then recompiled, but some of them just cannot be built outside of the ghc compilation process, and, AFAIU, this is exactly the purpose of multiple outputs.

The original use case was for splitting up /include from /lib. I don't think anyone has ever generated them from a variable list like this before. There's probably no technical reason why it can't be done, it is just a little bit unexpected. A lot of this can probably be solved by better tooling - I'm just not sure if we are ready for something like this. A lot of users are surprised when something like this doesn't work:

nix-env -iA ghc.base

(just ghc.out is installed IIRC)

kirelagin · 2018-07-11T17:19:09Z

Well, those multiple outputs are just a technical detail, it is actually a good thing that there is no easy way to discover them. They are intended to be used only as top-level haskell packages set attributes.

Ericson2314 · 2018-07-11T18:45:01Z

Another concern is that more invasive the procedure gets, The move convenient it is to have two derivations for purpose of debug cycles. Kind of like how we don't wrap gcc when be build it but separately.

This moves core libraries in Haskell to its own derivations. This is needed to avoid references to the GHC compiler ending up in binaries built by GHC. Instead of referencing GHC directly, these binaries will now link to the copies. Unfortunately, for this to work, we need to know which GHC libraries exist at evaluation time. This requires maintaining a list of the builtin library names & their versions in the GHC compiler’s “libs” attribute. A better way of handling this is needed.

ghc8.2.2 apparently doesn’t like moving rts from its original location. Hopefully this was fixed in ghc 8.6.1.

This may not be needed but it is more consistent with the idea of treating core libraries as ordinary haskell libraries.

Move score libs to pkgs/development/haskell-modules/default.nix. This seems like a better place for them. Instead of doing “base = null;” we now can use the actual base here.

matthewbauer · 2018-07-28T23:08:53Z

I honestly have no idea how to fix the unknown package: rts. Any debugging points to it being some sort of Cabal thing. I filed an issue here: haskell/cabal#5445. Does anyone else have experience with using -no-global-package-db?

Ericson2314 · 2018-07-31T04:30:24Z

pkgs/development/haskell-modules/default.nix

+
+      eval fixupPhase
+    '';
+  }; in builtins.listToAttrs (map (mkCoreLib ghc) (super.ghc.libraries or []));


Indentation?

Ericson2314 · 2018-07-31T14:30:32Z

pkgs/development/compilers/ghc/8.6.1.nix

@@ -208,6 +208,45 @@ stdenv.mkDerivation (rec {

    # Our Cabal compiler name
    haskellCompilerName = "ghc-8.4.3";


Unrelated issue here, btw

cdepillabout · 2020-05-05T05:40:54Z

@matthewbauer This PR is quite old at this point, is there anything in particular that needs to be done to move it forward?

Or could we go ahead and close it?

matthewbauer · 2020-05-09T18:36:24Z

Yes closing for now. It may be useful in the future but not in the current state

matthewbauer requested a review from peti as a code owner July 10, 2018 19:16

matthewbauer requested review from Ericson2314 and removed request for peti July 10, 2018 19:16

GrahamcOfBorg added the 6.topic: haskell label Jul 10, 2018

matthewbauer requested a review from peti July 10, 2018 19:24

matthewbauer changed the title ~~[wip] haskell: add core libraries logic to remove libraries from GHC~~ [wip] haskell: add core libraries logic to remove need for global package db Jul 10, 2018

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 10, 2018

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 and removed 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 10, 2018

kirelagin mentioned this pull request Jul 11, 2018

haskell: Split boot libraries into individual outputs (take 2) #43070

Closed

GrahamcOfBorg added the 2.status: merge conflict label Jul 18, 2018

matthewbauer and others added 4 commits July 18, 2018 15:30

haskell: use ghc861 for core libraries

d8c1f19

ghc8.2.2 apparently doesn’t like moving rts from its original location. Hopefully this was fixed in ghc 8.6.1.

haskell: put core libs in propagated build inputs

d1e4904

This may not be needed but it is more consistent with the idea of treating core libraries as ordinary haskell libraries.

haskell: cleanup core libs

4134557

Move score libs to pkgs/development/haskell-modules/default.nix. This seems like a better place for them. Instead of doing “base = null;” we now can use the actual base here.

matthewbauer force-pushed the ghc-core-libs branch from f6ca623 to 4134557 Compare July 18, 2018 19:31

GrahamcOfBorg removed the 2.status: merge conflict label Jul 18, 2018

matthewbauer mentioned this pull request Jul 18, 2018

Cabal can't configure without global package db haskell/cabal#5445

Closed

haskell: fix rts case

563b9e0

matthewbauer requested review from basvandijk and ryantm as code owners July 30, 2018 17:18

Ericson2314 reviewed Jul 31, 2018

View reviewed changes

matthewbauer mentioned this pull request Sep 6, 2018

Difficult to override the Cabal version used for setup.hs #46166

Open

domenkozar mentioned this pull request Sep 11, 2018

[Help request] Minimize the size of the nar files cachix/cachix#125

Closed

matthewbauer self-assigned this Dec 10, 2018

ryantm added the 2.status: work-in-progress label Feb 24, 2019

matthewbauer closed this Feb 27, 2019

matthewbauer reopened this Oct 19, 2019

ofborg bot added the 2.status: merge conflict label Oct 19, 2019

matthewbauer closed this May 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] haskell: add core libraries logic to remove need for global package db #43300

[wip] haskell: add core libraries logic to remove need for global package db #43300

matthewbauer commented Jul 10, 2018 •

edited

peti commented Jul 10, 2018

matthewbauer commented Jul 10, 2018 •

edited

matthewbauer commented Jul 10, 2018

Ericson2314 commented Jul 11, 2018 •

edited

domenkozar commented Jul 11, 2018

kirelagin commented Jul 11, 2018

kirelagin commented Jul 11, 2018

kirelagin commented Jul 11, 2018

Ericson2314 commented Jul 11, 2018

matthewbauer commented Jul 11, 2018 •

edited

kirelagin commented Jul 11, 2018

Ericson2314 commented Jul 11, 2018

matthewbauer commented Jul 28, 2018

Ericson2314 Jul 31, 2018

Ericson2314 Jul 31, 2018

cdepillabout commented May 5, 2020

matthewbauer commented May 9, 2020

		@@ -208,6 +208,45 @@ stdenv.mkDerivation (rec {

		# Our Cabal compiler name
		haskellCompilerName = "ghc-8.4.3";

[wip] haskell: add core libraries logic to remove need for global package db #43300

[wip] haskell: add core libraries logic to remove need for global package db #43300

Conversation

matthewbauer commented Jul 10, 2018 • edited

peti commented Jul 10, 2018

matthewbauer commented Jul 10, 2018 • edited

matthewbauer commented Jul 10, 2018

Ericson2314 commented Jul 11, 2018 • edited

domenkozar commented Jul 11, 2018

kirelagin commented Jul 11, 2018

kirelagin commented Jul 11, 2018

kirelagin commented Jul 11, 2018

Ericson2314 commented Jul 11, 2018

matthewbauer commented Jul 11, 2018 • edited

kirelagin commented Jul 11, 2018

Ericson2314 commented Jul 11, 2018

matthewbauer commented Jul 28, 2018

Ericson2314 Jul 31, 2018

Choose a reason for hiding this comment

Ericson2314 Jul 31, 2018

Choose a reason for hiding this comment

cdepillabout commented May 5, 2020

matthewbauer commented May 9, 2020

matthewbauer commented Jul 10, 2018 •

edited

matthewbauer commented Jul 10, 2018 •

edited

Ericson2314 commented Jul 11, 2018 •

edited

matthewbauer commented Jul 11, 2018 •

edited