Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Somewhat saner cross-compiling through bootstrapping #22107

Merged
merged 14 commits into from Jan 24, 2017

Conversation

Ericson2314
Copy link
Member

#21915 got closed by mistake due to a force push. @shlevy I wrote docs so this should be ready to go unless we want someone to review the docs.

The docs have some explicit "to be written" sections. I consider that fine, and the docs rightfully WIP, because some stuff will change with future PRs and there is little point writing it just to rewrite it.

Ericson2314 and others added 14 commits January 24, 2017 11:37
[N.B., this package also applies to the commits that follow it in the same
PR.]

In most cases, buildPackages = pkgs so things work just as before. For
cross compiling, however, buildPackages is resolved as the previous
bootstrapping stage. This allows us to avoid the mkDerivation hacks cross
compiling currently uses today.

To avoid a massive refactor, callPackage will splice together both package
sets. Again to avoid churn, it uses the old `nativeDrv` vs `crossDrv` to do
so. So now, whether cross compiling or not, packages with get a `nativeDrv`
and `crossDrv`---in the non-cross-compiling case they are simply the same
derivation. This is good because it reduces the divergence between the
cross and non-cross dataflow. See `pkgs/top-level/splice.nix` for a comment
along the lines of the preceding paragraph, and the code that does this
splicing.

Also, `forceNativeDrv` is replaced with `forceNativePackages`. The latter
resolves `pkgs` unless the host platform is different from the build
platform, in which case it resolves to `buildPackages`. Note that the
target platform is not important here---it will not prevent
`forcedNativePackages` from resolving to `pkgs`.

--------

Temporarily, we make preserve some dubious decisions in the name of preserving
hashes:

Most importantly, we don't distinguish between "host" and "target" in the
autoconf sense. This leads to the proliferation of *Cross derivations
currently used. What we ought to is resolve native deps of the cross "build
packages" (build = host != target) package set against the "vanilla
packages" (build = host = target) package set. Instead, "build packages"
uses itself, with (informally) target != build in all cases.

This is wrong because it violates the "sliding window" principle of
bootstrapping stages that shifting the platform triple of one stage to the
left coincides with the next stage's platform triple. Only because we don't
explicitly distinguish between "host" and "target" does it appear that the
"sliding window" principle is preserved--indeed it is over the reductionary
"platform double" of just "build" and "host/target".

Additionally, we build libc, libgcc, etc in the same stage as the compilers
themselves, which is wrong because they are used at runtime, not build
time. Fixing this is somewhat subtle, and the solution and problem will be
better explained in the commit that does fix it.

Commits after this will solve both these issues, at the expense of breaking
cross hashes. Native hashes won't be broken, thankfully.

--------

Did the temporary ugliness pan out? Of the packages that currently build in
`release-cross.nix`, the only ones that have their hash changed are
`*.gcc.crossDrv` and `bootstrapTools.*.coreutilsMinimal`. In both cases I
think it doesn't matter.

 1. GCC when doing a `build = host = target = foreign` build (maximally
    cross), still defines environment variables like `CPATH`[1] with
    packages.  This seems assuredly wrong because whether gcc dynamically
    links those, or the programs built by gcc dynamically link those---I
    have no idea which case is reality---they should be foreign. Therefore,
    in all likelihood, I just made the gcc less broken.

 2. Coreutils (ab)used the old cross-compiling infrastructure to depend on
    a native version of itself. When coreutils was overwritten to be built
    with fewer features, the native version it used would also be
    overwritten because the binding was tight. Now it uses the much looser
    `BuildPackages.coreutils` which is just fine as a richer build dep
    doesn't cause any problems and avoids a rebuild.

So, in conclusion I'd say the conservatism payed off. Onward to actually
raking the muck in the next PR!

[1]: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
The long term goal is a big replace:
  { inherit system platform; } => buildPlatform
  crossSystem => hostPlatform
  stdenv.cross => targetPlatform
And additionally making sure each is defined even when not cross compiling.

This commit refactors the bootstrapping code along that vision, but leaves
the old identifiers with their null semantics in place so packages can be
modernized incrementally.
…form`

This is an implementation detail of how the bootstrapping chain is chosen,
and thus need not be in the manual.
Do so in the stdenv section where the other two are discussed. This can be
done without brining up cross-compilation by talking about build-time vs
run-time.
@mention-bot
Copy link

@Ericson2314, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @vcunat, @viric, @LnL7 and @copumpkin to be potential reviewers.

@shlevy
Copy link
Member

shlevy commented Jan 24, 2017

@Ericson2314 Besides docs has anything changed since my last review?

@Ericson2314
Copy link
Member Author

@shlevy just docs and a tiny bit of comment formatting.

@Ericson2314
Copy link
Member Author

Ericson2314 commented Jan 24, 2017

Travis is backlogged, but I ran (the normal) tests locally and they pass. https://hydra.nixos.org/jobset/nixpkgs/sonarpulse-cross-tepid also checks out (no regressions).

@shlevy
Copy link
Member

shlevy commented Jan 24, 2017

:LGTM:

@Ericson2314 Ericson2314 merged commit 5ad696b into NixOS:master Jan 24, 2017
@Ericson2314 Ericson2314 deleted the cross-tepid branch January 24, 2017 20:12
shlevy referenced this pull request Feb 6, 2017
This makes the disk usage footprint of building the kernel smaller in 3
ways:

1) There is no separate kernel source derivation
2) Rather than using the entire build tree, only the output of make
modules_prepare is kept in the $dev output (plus the module symbol
versioning file generated during the build)
3) Only the subset of the source tree known to be needed for external
builds is kept in $dev

Note that while 2) is supported by official kernel documentation, I
couldn't find any source describing what we need to keep for 3). I've
started with the bare minimum (the main Makefile is called by the
Makefile generated by make modules_prepare) and we can/should add more
as needed for kernelPackages.

Signed-off-by: Shea Levy <shea@shealevy.com>
Ericson2314 added a commit to Ericson2314/nixpkgs that referenced this pull request Feb 7, 2017
…gh bootstrapping

Merge commit '5ad696b06763b7c5c6e1aa7ed848a50bff78c8d5' into release-cross-darwin
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants