WIP: EDK2, OVMF.fd: fixes for cross-compilation + dependencies + armv7l #53064

samueldr · 2018-12-30T05:50:10Z

THIS NEEDS FIXES CURRENTLY IN STAGING TO BUILD. It is why this is based on staging.

This also needs a good review by the peeps that know more about cross-compilation. Some of the fixes, especially the python-touching fixes, may not be optimal.

Motivation for this change

Well, get the OVMF EFI "floppy disks" (that's what .fd stands for?) for aarch64 and armv7l buildable with cross-compilation. Furthermore, get the armv7l build going too.

This is prep work for personal stuff, but hey, nixpkgs gets better here!

This was verified to work using qemu-system-* and the x86_64, aarch64 and armv7l NixOS iso. (The NixOS armv7l iso is another WIP thing, but it's really not good now.)

Things done

✔️ Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- ✔️ NixOS
- ⬜ macOS
- ⬜ other Linux distributions
⬜ Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
⬜ Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
✔️ Tested execution of all binary files (usually in ./result/bin/)
⬜ Determined the impact on package closure size (by running nix path-info -S before and after)
✔️ Assured whether relevant documentation is up to date
✔️ Fits CONTRIBUTING.md.

pkgs/development/compilers/edk2/default.nix

matthewbauer · 2018-12-30T06:12:12Z

pkgs/development/compilers/edk2/default.nix

+# Given a stdenv, returns the edk2-valid arch.
+envToArch = env:
+  if env.isi686 then
+    "IA32"


These keep showing up. We probably should put them in lib/systems/ as uefiShortName or something to avoid the duplication.

They do come often in similar sounding names, but all-caps they're pretty much the only instances :/. These ones, though, seem to be really specific to their build system.

iso-image.nix has those, which are not entirely equivalent, even ignoring case.

363 # Name used by UEFI for architectures. 364 targetArch = 365 if pkgs.stdenv.isi686 then 366 "ia32" 367 else if pkgs.stdenv.isx86_64 then 368 "x64" 369 else if pkgs.stdenv.isAarch64 then 370 "aa64" 371 else 372 throw "Unsupported architecture";

~~Should still be a case on env.parsed.cpu.name.~~ Nevermind. It's good as is, because the 32-bit ones are families of platform so there's no single string to case on.

Ericson2314 · 2019-01-02T05:20:16Z

pkgs/applications/virtualization/OVMF/default.nix

+
+  buildFlags = if stdenv.isAarch64 then ""
+    else if seabios == null then ''${lib.optionalString secureBoot "-DSECURE_BOOT_ENABLE=TRUE"}''
+    else ''-D CSM_ENABLE -D FD_SIZE_2MB ${lib.optionalString secureBoot "-DSECURE_BOOT_ENABLE=TRUE"}'';
 in

 stdenv.mkDerivation (edk2.setup projectDscPath {


I see below (cannot comment there) buildInputs = [nasm iasl] which is definitely wrong. (Those compilers should be nativeBuildInputs.) Try to cross compile this derivation too---building firmware should always be cross compilation anyways.

Yeah, GitHub may have eaten a previous reply I wrote. It worked mostly due to luck. I never cross-compiled to x86_64 from a non-x86_64 platform, making that nasm work. As for iasl, it was in the nativeBuildInputs from edk2.setup masking this issue.

For every time I push, the code has been tested to cross-compile and run for aarch64, and natively build on x86_64.

pkgs/development/compilers/edk2/default.nix

Ericson2314 · 2019-01-02T05:28:45Z

pkgs/development/compilers/edk2/default.nix

-      buildInputs = [ pythonEnv ] ++
-        stdenv.lib.optionals (attrs ? buildInputs) attrs.buildInputs;
+      buildInputs = [] ++ attrs.buildInputs or [];
+      nativeBuildInputs = [ buildPythonEnv buildPackages.iasl ] ++ attrs.nativeBuildInputs or [];


Can do iasl instead of buildPackages.iasl here because of the splicing black magic. (See https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/splice.nix .)

Actually, it's even more complex because the package using the edk should always live in a different bootstrapping phase (hence my suggestion above that OVMF should use buildPackages.edk2). This means that this setup really ought to be defined elsewhere so that anything used here can come from the phase of things using edk, rather than the phase providing edk. Otherwise all the splicing stuff / manual buildPackages / targetPackages will make no sense.

What I just wrote probably in general makes no sense :). But it is exactly analogous how stdenv provided tools from the previous stage and lives in the same stage as things that use it, rather than being defined in the previous stage with the tools, and being used in the next stage. setup is a wrapper around stdenv.mkDerivation, so just as stdenv use buildPackages.gcc, so should setup use buildPackages.edk2. But then it makes little sense to put as a passthru on edk2 since then we spuriously reference edk to get setup which just uses buildPackages.edk + all the splicing issues I referenced above.

pkgs/applications/virtualization/OVMF/default.nix

Ericson2314 · 2019-01-02T05:34:03Z

pkgs/applications/virtualization/OVMF/default.nix

+
+    build \
+      -n $NIX_BUILD_CORES \
+      ${buildFlags} \


Can you use escapeShellArgs to make this an array instead?

I won't right now, I don't think this PR will ever be merged as-is, see the other comment where I explain I'm ending up reworking the edk2.setup thing to work better and in a more generic fashion.

Though, I will on that branch.

pkgs/development/compilers/edk2/default.nix

Ericson2314 · 2019-01-02T05:49:10Z

pkgs/development/compilers/edk2/default.nix

+pythonEnv = python2.withPackages(ps: [ps.tkinter]);
+
+targetArch = envToArch targetPlatform;
+hostArch = envToArch buildPlatform;


Should be hostPlatform. It's not a good idea to get tricky and rename these!

Ericson2314 · 2019-01-02T05:52:22Z

In general, getting this stuff right because as far as I can tell from a cursory look at the docs, the EDK 2 is trying to be package manager for cross-compilation, and doing a poor job of it. The best results would probably the larger project of ripping it shreds and replacing as much stuff with nixpkgs packages as possible, and exposing the new chunks in some sort of edk-packages or tianocore-packages.

samueldr · 2019-01-02T06:17:18Z

In another branch, building-up from the cross-compilation branch, I'm working on making the EDK2 stuff more re-usable through making use of it to build a Raspberry Pi 3 UEFI implementation. I don't think I'm going as far as you're suggesting, but I think the end-result is better and easier to follow along. Though, this makes fixing things here annoying as some of the things are going to be removed :).

Thanks for the review, it really helps get those things together.

Ericson2314 · 2019-01-02T15:43:12Z

OK great! Happy to help with that one too.

FRidh · 2019-01-02T19:16:53Z

I've taken some of your Python-related changes and fixed cross-compilation of Python packages in #53123. At least, it mostly seems to work. numpy not yet.

FRidh · 2019-04-28T08:04:40Z

@samueldr are you still working on this? If so, please rebase.

samueldr · 2019-04-28T15:34:22Z

I intend to work on this, haven't for a while, will rebase at that moment.

If others are interested in working on this be my guest!

stale · 2020-06-02T02:47:54Z

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the
related code and @ mention them in a comment.
Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
irc.freenode.net.

Mic92 · 2020-06-02T09:54:10Z

I converted it to a draft.

stale · 2021-06-09T00:40:31Z

I marked this as stale due to inactivity. → More info

samueldr · 2022-12-23T03:42:46Z

I'm not coming back to this exact PR ever again, so I'm closing it.

samueldr added 3 commits December 30, 2018 00:44

iasl: fixes for cross compiling

5a3774f

setuptools: fixes for cross compiling

6d0f7de

mkPythonDerivation: fixes for cross compiling

2c0e50c

samueldr requested review from matthewbauer, Ericson2314, FRidh and Mic92 December 30, 2018 05:50

GrahamcOfBorg added the 6.topic: python label Dec 30, 2018

matthewbauer reviewed Dec 30, 2018

View reviewed changes

pkgs/development/compilers/edk2/default.nix Outdated Show resolved Hide resolved

GrahamcOfBorg added 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Dec 30, 2018

matthewbauer reviewed Dec 30, 2018

View reviewed changes

samueldr added 2 commits December 30, 2018 01:17

edk2: Misc cross-compilation fixes

34ba7dd

OVMF.fd: Misc cross-compilation fixes

f49e539

samueldr force-pushed the tianocore/cross branch from d2c27d5 to f49e539 Compare December 30, 2018 06:17

samueldr requested review from alyssais, basvandijk, edolstra, infinisil, LnL7, nbp, peti, ryantm, ttuegel and zimbatm as code owners December 30, 2018 06:17

samueldr changed the base branch from master to staging December 30, 2018 06:18

samueldr removed request for zimbatm, ryantm, peti and ttuegel December 30, 2018 06:20

Ericson2314 reviewed Jan 2, 2019

View reviewed changes

pkgs/development/compilers/edk2/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 2, 2019

View reviewed changes

pkgs/development/compilers/edk2/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 2, 2019

View reviewed changes

pkgs/applications/virtualization/OVMF/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 2, 2019

View reviewed changes

pkgs/development/compilers/edk2/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 2, 2019

View reviewed changes

samueldr added 5 commits January 2, 2019 01:18

WIP: prefer using stdenv.cc.targetPrefix.

37c8b55

WIP: removes useless buildInputs.

1aa499a

WIP

0fefe26

WIP: fixes build inputs for OVMF and reduces dead code.

5cd8d8e

WIP: resolves nativeBuildInputs issues.

e70532c

ryantm added the 2.status: work-in-progress label Feb 24, 2019

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 2, 2020

Mic92 marked this pull request as draft June 2, 2020 09:53

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 2, 2020

ryantm added the 2.status: merge conflict label Oct 3, 2020

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 9, 2021

samueldr closed this Dec 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: EDK2, OVMF.fd: fixes for cross-compilation + dependencies + armv7l #53064

WIP: EDK2, OVMF.fd: fixes for cross-compilation + dependencies + armv7l #53064

samueldr commented Dec 30, 2018 •

edited

matthewbauer Dec 30, 2018

samueldr Dec 30, 2018

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019 •

edited

samueldr Jan 2, 2019

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019

samueldr Jan 2, 2019

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 commented Jan 2, 2019 •

edited

samueldr commented Jan 2, 2019

Ericson2314 commented Jan 2, 2019

FRidh commented Jan 2, 2019

FRidh commented Apr 28, 2019

samueldr commented Apr 28, 2019

stale bot commented Jun 2, 2020

Mic92 commented Jun 2, 2020

stale bot commented Jun 9, 2021

samueldr commented Dec 23, 2022

WIP: EDK2, OVMF.fd: fixes for cross-compilation + dependencies + armv7l #53064

WIP: EDK2, OVMF.fd: fixes for cross-compilation + dependencies + armv7l #53064

Conversation

samueldr commented Dec 30, 2018 • edited

Motivation for this change

Things done

matthewbauer Dec 30, 2018

Choose a reason for hiding this comment

samueldr Dec 30, 2018

Choose a reason for hiding this comment

Ericson2314 Jan 2, 2019 • edited

Choose a reason for hiding this comment

Ericson2314 Jan 2, 2019 • edited

Choose a reason for hiding this comment

samueldr Jan 2, 2019

Choose a reason for hiding this comment

Ericson2314 Jan 2, 2019 • edited

Choose a reason for hiding this comment

Ericson2314 Jan 2, 2019 • edited

Choose a reason for hiding this comment

Ericson2314 Jan 2, 2019

Choose a reason for hiding this comment

samueldr Jan 2, 2019

Choose a reason for hiding this comment

Ericson2314 Jan 2, 2019 • edited

Choose a reason for hiding this comment

Ericson2314 commented Jan 2, 2019 • edited

samueldr commented Jan 2, 2019

Ericson2314 commented Jan 2, 2019

FRidh commented Jan 2, 2019

FRidh commented Apr 28, 2019

samueldr commented Apr 28, 2019

stale bot commented Jun 2, 2020

Mic92 commented Jun 2, 2020

stale bot commented Jun 9, 2021

samueldr commented Dec 23, 2022

samueldr commented Dec 30, 2018 •

edited

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 Jan 2, 2019 •

edited

Ericson2314 commented Jan 2, 2019 •

edited