kernel: allow to be built on remote builder #105675

Mic92 · 2020-12-02T09:32:50Z

When building a kernel for a different architecture, the build
fails because preferLocalBuild tries to run the wrong binaries
locally (i.e. aarch64 binaries on x86-64).

Motivation for this change

Things done

When building a kernel for a different architecture, the build fails because `preferLocalBuild` tries to run the wrong binaries locally (i.e. aarch64 binaries on x86-64).

Mic92 · 2020-12-02T09:33:53Z

Tested with

$ nix-build -A linux_ayufan_5_9 --system 'aarch64-linux' -j 0

in https://github.com/Mic92/nur-packages/

Mic92 · 2020-12-02T09:34:13Z

not sure if there is a better way?

Mic92 · 2020-12-02T09:35:11Z

cc @cleverca22

layus

I think that for such a small change scoped in a technical part of nixpkgs this is okay.

But this comes with several unanswered questions.

Using overrides here is wrong. If there are valid cases for flipping preferLocalBuild and allowSubstitutes in writeTextFile, then we should make it configurable in writeTextFile.
It is unclear why this code prevents the remote build of cross kernel builds. Is it because
a. it comes with stdev.shell that has the wrong architecture and cannot execute remotely ?
b. It also contains allowSubstitutes=false which prevents nix from uploading that file on the remote server ?
c. Nix plays badly with the mix of options anyway.
There appears to be some mis-classification iof this script. It appears to be a nativeBuildInput, while the way it is used (an antiquotation) makes it a normal build input by default. Could the problem come from there instead ?

Please forgive my lack of knowledge on remote cross compilation if need be. But I think there is an opportunity to gain some knowledge here. An possibly generalize the issue.

andir · 2020-12-02T11:50:53Z

I am not sure this change is required at all. I've been remote building kernels for my aarch64 machines for a while now. The build machine doesn't have the binfmt setup so whenever it has to build aarch64 derivations it does so on a remote machine.

What error did you run into without this change and how did you setup the remote builders?

Mic92 · 2020-12-02T19:12:52Z

I am not sure this change is required at all. I've been remote building kernels for my aarch64 machines for a while now. The build machine doesn't have the binfmt setup so whenever it has to build aarch64 derivations it does so on a remote machine.

What error did you run into without this change and how did you setup the remote builders?

$ nix-build -A linux_ayufan_5_9 --system 'aarch64-linux' -j 0
these 2 derivations will be built:
  /nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv
  /nix/store/pzdgpi52xf7d1v8jmhnav53axrlq538d-linux-5.9.0-1145-ayufan.drv
building '/nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv'...

ErroSysErrorvexecuting '/nix/store/g2xbi9c7gxrq8d2gmpmyx39fb8x5wkwb-bash-4.4-p23/bin/bash': Exec format error
error: --- Error --- nix-daemon
builder for '/nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv' failed with exit code 1; last 2 log lines:

  ErroSysErrorvexecuting '/nix/store/g2xbi9c7gxrq8d2gmpmyx39fb8x5wkwb-bash-4.4-p23/bin/bash': Exec format error
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------- nix-build
1 dependencies of derivation '/nix/store/pzdgpi52xf7d1v8jmhnav53axrlq538d-linux-5.9.0-1145-ayufan.drv' failed to build
$ nix log /nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ ^A
   2   │ ^E^@^@^@^@^@^@^@Error^@^@^@^@^@^@^@^@^@^@^@^H^@^@^@^@^@^@^@SysError^@^@^@^@^@^@^@^@v^@^@^@^@^@^@^@executing '/nix/store/g2xbi9c7gxrq8d2gmpmyx39fb8x5wkwb-bash-4.4-p23/bin/bash': Exec format error^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Mic92 · 2020-12-02T19:14:15Z

This my nix config:

# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
build-users-group = nixbld
max-jobs = auto
cores = 0
sandbox = true



substituters = https://nix-community.cachix.org https://mic92.cachix.org https://cache.nixos.org/
trusted-substituters = 
trusted-public-keys = nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs= mic92.cachix.org-1:gi8IhgiT3CYZnJsaW7fxznzTkMUOn1RY4GmXdT/nXYQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
auto-optimise-store = false
require-sigs = true
trusted-users = nix joerg root
allowed-users = *

system-features = nixos-test benchmark big-parallel kvm
sandbox-fallback = false

builders-use-substitutes = true
keep-outputs = true
keep-derivations = true
# in zfs we trust
fsync-metadata = false
experimental-features = nix-command flakes

flake-registry = /nix/store/phbpq3yxyf96kilx3k9yaxlkyjwcqr01-source/flake-registry.json

# /etc/nix/machines
nix@martha.r x86_64-linux /run/secrets/id_buildfarm 8 1 big-parallel,kvm,nixos-test 
nix@donna.r x86_64-linux /run/secrets/id_buildfarm 8 1 big-parallel,kvm,nixos-test 
nix@amy.r x86_64-linux /run/secrets/id_buildfarm 8 1 big-parallel,kvm,nixos-test 
nix@clara.r x86_64-linux /run/secrets/id_buildfarm 8 1 big-parallel,kvm,nixos-test 
nix@rose.r x86_64-linux /run/secrets/id_buildfarm 8 1 big-parallel,kvm,nixos-test 
mic92@prism.r x86_64-linux /root/.ssh/id_ed25519 4 1 big-parallel,kvm,nixos-test 
nix@eve.thalheim.io x86_64-linux /run/secrets/id_buildfarm 4 1 big-parallel,kvm,nixos-test 
mic92@aarch64.nixos.community aarch64-linux /root/.ssh/id_ed25519 96 1 big-parallel,kvm,nixos-test

andir · 2020-12-03T11:06:09Z

I am not sure this change is required at all. I've been remote building kernels for my aarch64 machines for a while now. The build machine doesn't have the binfmt setup so whenever it has to build aarch64 derivations it does so on a remote machine.
What error did you run into without this change and how did you setup the remote builders?

$ nix-build -A linux_ayufan_5_9 --system 'aarch64-linux' -j 0
these 2 derivations will be built:
  /nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv
  /nix/store/pzdgpi52xf7d1v8jmhnav53axrlq538d-linux-5.9.0-1145-ayufan.drv
building '/nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv'...

ErroSysErrorvexecuting '/nix/store/g2xbi9c7gxrq8d2gmpmyx39fb8x5wkwb-bash-4.4-p23/bin/bash': Exec format error
error: --- Error --- nix-daemon
builder for '/nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv' failed with exit code 1; last 2 log lines:

  ErroSysErrorvexecuting '/nix/store/g2xbi9c7gxrq8d2gmpmyx39fb8x5wkwb-bash-4.4-p23/bin/bash': Exec format error
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------- nix-build
1 dependencies of derivation '/nix/store/pzdgpi52xf7d1v8jmhnav53axrlq538d-linux-5.9.0-1145-ayufan.drv' failed to build
$ nix log /nix/store/ry84gwy7ypqj6qym3k0y1jvadcajnbpn-installkernel.drv
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ ^A
   2   │ ^E^@^@^@^@^@^@^@Error^@^@^@^@^@^@^@^@^@^@^@^H^@^@^@^@^@^@^@SysError^@^@^@^@^@^@^@^@v^@^@^@^@^@^@^@executing '/nix/store/g2xbi9c7gxrq8d2gmpmyx39fb8x5wkwb-bash-4.4-p23/bin/bash': Exec format error^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

By setting --system … you are telling Nix that your local machine is that system type. What you really want is to pass the correct system into the nixpkgs invocation so that the derivations will have system = "aarch64-linux"; while your local machine is still x86_64-linux (or whatever). Then the remote builder logic works. --system really isn't thought for these kinds of things. EDIT: I think --system did make sense when x86_64-linux and i688-linux were common to be used on the same system.

Apply this patch to your nur repo:

diff --git a/default.nix b/default.nix
index c7aa03f..43819be 100644
--- a/default.nix
+++ b/default.nix
@@ -1,4 +1,4 @@
-{ pkgs ? import <nixpkgs> { } }:
+{ system ? builtins.currentSystem,  pkgs ? import <nixpkgs> { inherit system; } }:
 
 rec {
   bitwarden_rs_ldap = pkgs.callPackage ./pkgs/bitwarden_rs_ldap { };

And then build like this: nix-build -A linux_ayufan_5_6 --argstr system "aarch64-linux"

Mic92 · 2020-12-03T11:09:32Z

Thanks!

kernel: allow to be built on remote builder

ada0589

When building a kernel for a different architecture, the build fails because `preferLocalBuild` tries to run the wrong binaries locally (i.e. aarch64 binaries on x86-64).

ofborg bot requested review from teto, Atemu, thoughtpolice and layus December 2, 2020 09:42

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 501+ 10.rebuild-linux: 1001-2500 labels Dec 2, 2020

layus reviewed Dec 2, 2020

View reviewed changes

Mic92 closed this Dec 3, 2020

Mic92 deleted the fix-kernel-cross branch December 3, 2020 11:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kernel: allow to be built on remote builder #105675

kernel: allow to be built on remote builder #105675

Mic92 commented Dec 2, 2020 •

edited

Mic92 commented Dec 2, 2020

Mic92 commented Dec 2, 2020

Mic92 commented Dec 2, 2020

layus left a comment

andir commented Dec 2, 2020

Mic92 commented Dec 2, 2020

Mic92 commented Dec 2, 2020 •

edited

andir commented Dec 3, 2020 •

edited

Mic92 commented Dec 3, 2020

kernel: allow to be built on remote builder #105675

kernel: allow to be built on remote builder #105675

Conversation

Mic92 commented Dec 2, 2020 • edited

Motivation for this change

Things done

Mic92 commented Dec 2, 2020

Mic92 commented Dec 2, 2020

Mic92 commented Dec 2, 2020

layus left a comment

Choose a reason for hiding this comment

andir commented Dec 2, 2020

Mic92 commented Dec 2, 2020

Mic92 commented Dec 2, 2020 • edited

andir commented Dec 3, 2020 • edited

Mic92 commented Dec 3, 2020

Mic92 commented Dec 2, 2020 •

edited

Mic92 commented Dec 2, 2020 •

edited

andir commented Dec 3, 2020 •

edited