New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Somewhat saner cross-compiling through bootstrapping #22107
Conversation
[N.B., this package also applies to the commits that follow it in the same PR.] In most cases, buildPackages = pkgs so things work just as before. For cross compiling, however, buildPackages is resolved as the previous bootstrapping stage. This allows us to avoid the mkDerivation hacks cross compiling currently uses today. To avoid a massive refactor, callPackage will splice together both package sets. Again to avoid churn, it uses the old `nativeDrv` vs `crossDrv` to do so. So now, whether cross compiling or not, packages with get a `nativeDrv` and `crossDrv`---in the non-cross-compiling case they are simply the same derivation. This is good because it reduces the divergence between the cross and non-cross dataflow. See `pkgs/top-level/splice.nix` for a comment along the lines of the preceding paragraph, and the code that does this splicing. Also, `forceNativeDrv` is replaced with `forceNativePackages`. The latter resolves `pkgs` unless the host platform is different from the build platform, in which case it resolves to `buildPackages`. Note that the target platform is not important here---it will not prevent `forcedNativePackages` from resolving to `pkgs`. -------- Temporarily, we make preserve some dubious decisions in the name of preserving hashes: Most importantly, we don't distinguish between "host" and "target" in the autoconf sense. This leads to the proliferation of *Cross derivations currently used. What we ought to is resolve native deps of the cross "build packages" (build = host != target) package set against the "vanilla packages" (build = host = target) package set. Instead, "build packages" uses itself, with (informally) target != build in all cases. This is wrong because it violates the "sliding window" principle of bootstrapping stages that shifting the platform triple of one stage to the left coincides with the next stage's platform triple. Only because we don't explicitly distinguish between "host" and "target" does it appear that the "sliding window" principle is preserved--indeed it is over the reductionary "platform double" of just "build" and "host/target". Additionally, we build libc, libgcc, etc in the same stage as the compilers themselves, which is wrong because they are used at runtime, not build time. Fixing this is somewhat subtle, and the solution and problem will be better explained in the commit that does fix it. Commits after this will solve both these issues, at the expense of breaking cross hashes. Native hashes won't be broken, thankfully. -------- Did the temporary ugliness pan out? Of the packages that currently build in `release-cross.nix`, the only ones that have their hash changed are `*.gcc.crossDrv` and `bootstrapTools.*.coreutilsMinimal`. In both cases I think it doesn't matter. 1. GCC when doing a `build = host = target = foreign` build (maximally cross), still defines environment variables like `CPATH`[1] with packages. This seems assuredly wrong because whether gcc dynamically links those, or the programs built by gcc dynamically link those---I have no idea which case is reality---they should be foreign. Therefore, in all likelihood, I just made the gcc less broken. 2. Coreutils (ab)used the old cross-compiling infrastructure to depend on a native version of itself. When coreutils was overwritten to be built with fewer features, the native version it used would also be overwritten because the binding was tight. Now it uses the much looser `BuildPackages.coreutils` which is just fine as a richer build dep doesn't cause any problems and avoids a rebuild. So, in conclusion I'd say the conservatism payed off. Onward to actually raking the muck in the next PR! [1]: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
The long term goal is a big replace: { inherit system platform; } => buildPlatform crossSystem => hostPlatform stdenv.cross => targetPlatform And additionally making sure each is defined even when not cross compiling. This commit refactors the bootstrapping code along that vision, but leaves the old identifiers with their null semantics in place so packages can be modernized incrementally.
Semantics should be unchanged
…form` This is an implementation detail of how the bootstrapping chain is chosen, and thus need not be in the manual.
Do so in the stdenv section where the other two are discussed. This can be done without brining up cross-compilation by talking about build-time vs run-time.
@Ericson2314, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @vcunat, @viric, @LnL7 and @copumpkin to be potential reviewers. |
@Ericson2314 Besides docs has anything changed since my last review? |
@shlevy just docs and a tiny bit of comment formatting. |
Travis is backlogged, but I ran (the normal) tests locally and they pass. https://hydra.nixos.org/jobset/nixpkgs/sonarpulse-cross-tepid also checks out (no regressions). |
:LGTM: |
This makes the disk usage footprint of building the kernel smaller in 3 ways: 1) There is no separate kernel source derivation 2) Rather than using the entire build tree, only the output of make modules_prepare is kept in the $dev output (plus the module symbol versioning file generated during the build) 3) Only the subset of the source tree known to be needed for external builds is kept in $dev Note that while 2) is supported by official kernel documentation, I couldn't find any source describing what we need to keep for 3). I've started with the bare minimum (the main Makefile is called by the Makefile generated by make modules_prepare) and we can/should add more as needed for kernelPackages. Signed-off-by: Shea Levy <shea@shealevy.com>
…gh bootstrapping Merge commit '5ad696b06763b7c5c6e1aa7ed848a50bff78c8d5' into release-cross-darwin
#21915 got closed by mistake due to a force push. @shlevy I wrote docs so this should be ready to go unless we want someone to review the docs.
The docs have some explicit "to be written" sections. I consider that fine, and the docs rightfully WIP, because some stuff will change with future PRs and there is little point writing it just to rewrite it.