P.S. @Ericson2314 Redox OS can now be declaratively compiled using Nix! (warning: very long build time)

$ nix-build https://github.com/nix-community/redoxpkgs/archive/vmdisk.tar.gz -A pkgsCross.x86_64-unknown-redox.redox-vmdisk
$ cp result/harddisk.bin .
$ chmod +rw harddisk.bin
$ qemu-system-x86_64 -serial mon:stdio -d cpu_reset -d guest_errors -smp 4 -m 1024 -s -machine q35 -device ich9-intel-hda -device hda-duplex -net nic,model=e1000 -net user -device nec-usb-xhci,id=xhci -device usb-tablet,bus=xhci.0 -enable-kvm -cpu host -drive file=harddrive.bin,format=raw

The magic can be found at https://github.com/nix-community/redoxpkgs/tree/vmdisk/pkgs/redox-vmdisk

Sorry for not seeing this earlier! (Feel free to harass me on IRC, your work is very important to me :)).

So this is great, first of all. Really excited.

The specific thing I was talking about is a bit different, but I think will readily combine with this. I startied writing some code to explain what I mean:

diff --git a/pkgs/development/compilers/rust/default.nix b/pkgs/development/compilers/rust/default.nix
index d08b63dd643..32102b03be4 100644
--- a/pkgs/development/compilers/rust/default.nix
+++ b/pkgs/development/compilers/rust/default.nix
@@ -13,14 +13,36 @@
 , llvmPackages_5
 , pkgsBuildTarget, pkgsBuildBuild
 }: rec {
-  toRustTarget = platform: with platform.parsed; let
-    cpu_ = {
-      "armv7a" = "armv7";
-      "armv7l" = "armv7";
-      "armv6l" = "arm";
-    }.${cpu.name} or platform.rustc.arch or cpu.name;
-  in platform.rustc.config
-    or "${cpu_}-${vendor.name}-${kernel.name}${lib.optionalString (abi.name != "unknown") "-${abi.name}"}";
+  _rustTargetJson = platform:
+    builtins.removeAttrs (platform.rustc or {}) [ "name" ];
+
+  _toRustTargetJsonOpt = platform: let
+      targetJson = _rustTargetJson platform;
+      targetFileJsonContents = builtins.toJSON targetJson;
+      targetFileName = (platform.rustc.name or (builtins.hash "sha256" targetFileJsonContents)) + ".json";
+    in
+      if targetJson == {}
+      then null
+      else [ targetFileName targetFileJsonContents ];
+
+  toRustTarget = platform: let
+    noJsonDefaultTargetName = with platform.parsed; let
+      cpu_ = {
+        "armv7a" = "armv7";
+        "armv7l" = "armv7";
+        "armv6l" = "arm";
+      }.${cpu.name} or platform.rustc.arch or cpu.name;
+    in platform.rustc.config
+      or "${cpu_}-${vendor.name}-${kernel.name}${lib.optionalString (abi.name != "unknown") "-${abi.name}"}";
+    jsonAndNameOpt = _toRustTargetJsonOpt platform;
+  in
+    if jsonAndNameOpt == null
+    then noJsonDefaultTargetName
+    else builtins.head jsonAndNameOpt;
+
+  # Don't builtins.readFile this, else eval depends on derivation building
+  # which sadly is not allowed for nixpkgs.
+  toRustTargetTomlFileOpt = platform: ... (_toRustTargetJsonOpt platform) ...;
 
   makeRustPlatform = { rustc, cargo, ... }: rec {
     rust = {

So fill in toRustTargetTomlFileOpt with what you've got, following the comment, and then use this back in the other file.

Hopefully with NixOS/nix#3929 we can stop worrying about builtins.readFile and get rid of the heredoc, but I'm afraid that what you've got so far is largely unavoidable.

I suppose there is one alternative: keep everything in nix until you do one big toToml for everything. You can try that if you like, but I don't much care either way.

CC @cleverca22 & @Mic92. Looks like we are on the cusp of proper custom target cross-compilation support!

Hmm, I'm having trouble understanding that diff. Are you suggesting that the the target JSON be stored in stdenv.hostPlatform.rustc? I'd also like to note that the TOML code is only used for sysroot code (needed for building targets that rustc itself wasn't compiled for). TOML isn't used for the target JSON.

Either way, based on crossing paths with the work you've done with Rust for custom targets, you seem to be much more experienced with this than I.

Are you suggesting that the target JSON be stored in stdenv.hostPlatform.rustc?

Yup!

I'd also like to note that the TOML code is only used for sysroot code (needed for building targets that rustc itself wasn't compiled for).

Ah, I didn't realize that. So it is TOML for sysroot, and JSON for everything else?

Either way, based on crossing paths with the work you've done with Rust for custom targets, you seem to be much more experienced with this than I.

I'm really rusty with the details of target JSON etc, vs general idea / nixpkgs-specific stuff, so don't take me too seriously on the nitty-gritty! The basic ideas are:

• Let's try to specify all target-config-like stuff from stdenv.*Platform. platform.rustc is directly inspired by platform.gcc.
  • Always better to have structured values than files and strings
  • Because build.readFile ...derivation... and import ...derivation... are prohibited for stuff Nixpkgs' hydra builds, let's try to avoid "baking" config to files until as late as possible.
• Sysrooots are kinda evil, but definitely worth it for now until https://github.com/rust-lang/wg-cargo-std-aware wraps up.
  • Over time, with the work of that working group, standard library and even rustc itself should be built with buildRustPackage or buildRustCrate (or whatever the latest and great riff on those is).

You're doing great so far :), moving the config to stdenv.hostPlatform I don't think is a major deparature from what you've got.

Ah, I didn't realize that. So it is TOML for sysroot, and JSON for everything else?

Correct

You're doing great so far :), moving the config to stdenv.hostPlatform I don't think is a major deparature from what you've got.

Thanks! So I can figure out how to use this for Redox, how would I define a custom stdenv platform for the Redox kernel (which, unlike userspace, blacklists some instructions and lacks a libc)?

Redox userspace supports enough packages that I could create a PR to put that platform in Nixpkgs. The Redox kernel platform, however, would be used for nothing but the Redox kernel. Is there some way to define a stdenv.hostPlatform outside of Redox, then have that platform used for only one package (instead of needing to re-compile rustc etc)?

Yeah! Do something like

import <nixpkgs> {
  crossSystem = {
    config = "x86_64-unknown-none-none"; # something like this
    platform.rustc = {
      name = "bare-metal-for-redox-or-whatever-you-want-to-call-it";
      ...jsonContents...
    };
  };
  overlays = [ ... ];
}

If the redox platform is stable, you can put it in lib/systems/platforms.nix. Basically that file is a grab-bag of stuff awaiting to be better organized, so I donn't really care if niche things are in there.

C.f. mozilla/nixpkgs-mozilla#222. Eventually, it would be nice if mozilla-nixpkgs could work well with this stuff to allow one to seamlessly switch between prebuilt sysroots (original made for rust-up) and nix-built ones.

Oh awesome! Thanks for the explanation. I'll try to work on that this weekend :-)

Can we have also an derivation that exercise this feature? Rust releases happens quite quickly. We currently have a testplan that includes building firefox/thunderbird on Linux and fd on macOS. We could bundle those up in a attrset and also include a test for custom target so that we can have one command that we use for testing every update.

Can we have also an derivation that exercise this feature?

Yeah, that sounds like a good idea. Maybe add a derivation [1] with a new target (x86_64-unknown-blog_os)?

[1] https://github.com/phil-opp/blog_os/tree/first_edition_post_4

I feel like there should be some way to create a new pkgsCross without using import. Something like pkgsCross.fromCrossSystem { ... }

I've figured out a way to handle Rust targets having information both general to a runtime environment (e.g. CPU architecture) and specific to a project (e.g. project-specific compile flags). I've also written code for generating most of the target information from other attributes in stdenv.hostPlatform.

I'm having trouble putting this idea into words at the moment, but I'm currently waiting on [1] to see if it will work for x86_64

[1] https://users.rust-lang.org/t/why-is-there-no-default-x86-64-unknown-none-target/47825

I plan to push some fixes to this PR tomorrow or on Monday. Specifically, I plan to push code that uses Python to generate the sysroot Cargo.toml

EDIT: I'm still trying to find compute time to compile Rust, as mentioned on IRC. I have an idea, so I'll give it a shot

I've added a test for compiling blog_os and cleaned up a little. I put these in changes small commits so that they're easy to review.

I added assert useSysroot -> !(args.doCheck or true) and relevant documentation stating that tests won't work because custom sysroots are being built without the std crate (is this a blocking issue for this PR?).

I also pushed a commit (d884b2d) that sets the RUSTFLAGS environment variable as a derivation attribute instead of injecting it into buildPhase. This seems cleaner, but I'd like for the change to be reviewed since it's a notable last-minute change. I'd be happy to revert this commit if so desired.

P.S. Feel free to rename this PR and the final commit to Nixpkgs to buildRustPackage: support custom targets. I forgot where the original makeRustPlatform text in the title came from.

Note that the new update-lockfile.sh script doesn't update the cargoSha256.

FYI: I have the feeling rust did some breaking changes to sysroot. Not sure if this will affect this pr (some folders are renamed)

ugh I missed the test!

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/build-rust-app-using-cargos-build-std-feature-with-naersk-fails-because-rust-src-is-missing/13161/5