Support extra compatible architectures #1916

lheckemann · 2018-02-27T14:24:51Z

This is based on @cleverca22's patch (cleverca22/nix-misc). It allows building for additional architectures locally. This can be useful with binfmt-misc or compatible architectures (e.g. ARM32-capable aarch64 machines). Additionally, it enables 32-bit ARM seccomp support.

Questions:

Thoughts on disabling i686-linux on x86_64-linux by default?
What about the seccomp setting? I experienced crashes caused by seccomp while testing on an ARM32-capable aarch64 machine, but I'm not sure how this will behave on non-ARM32-capable ones — in case it fails, I didn't throw an error. I'm not sure if this is appropriate, feedback would be appreciated.

This allows specifying additional systems that a machine is able to build for. This may apply on some armv7-capable aarch64 processors, or on systems using qemu-user with binfmt-misc to support transparent execution of foreign-arch programs. This removes the previous hard-coded assumptions about which systems are ABI-compatible with which other systems, and instead relies on the user to specify any additional platforms that they have ensured compatibility for and wish to build for locally. NixOS should probably add i686-linux on x86_64-linux systems for this setting by default.

This allows building armv[67]l-linux derivations on compatible aarch64 machines. Failure to add the architecture may result from missing hardware support, in which case we can't run 32-bit binaries and don't need to restrict them with seccomp anyway,

shlevy

I'm lukewarm on this. Partly because it seems like kind of a niche use case, and partly because I don't really trust qemu-user enough (though that's a gut feeling that's not well-founded). I'm open to be swayed if there are compelling use cases.

Either way needs docs, ideally with pointers to how you might use this with qemu-user

shlevy · 2018-02-27T14:28:08Z

src/libstore/globals.hh

@@ -295,6 +295,9 @@ public:
        "Nix store has a valid signature (that is, one signed using a key "
        "listed in 'trusted-public-keys'."};

+    Setting<StringSet> extraPlatforms{this, StringSet{}, "build-extra-platforms",


IMO this should default ti i686-linux on x86_64-linux

Makes sense.

Shouldn't we have all the entries from from https://github.com/NixOS/nix/pull/1916/files#diff-81a5a969aed0f3f9d6dc4e99b92925cdL60 ?

I'm not sure having all of them is a good idea, given how unreliable actually building them is: building armv6 stuff on an armv7 machine will often result in code that doesn't actually run on armv6 because of annoying autodetection in various build systems. Users who want to risk that can easily set this setting as appropriate.

bjornfor · 2018-02-27T17:43:58Z

partly because I don't really trust qemu-user enough (though that's a gut feeling that's not well-founded).

That is my impression too. Relevant: NixOS/nixpkgs#33553.

lheckemann · 2018-02-28T16:48:33Z

Oh yes, the docs. Whoops :)

As for its actual use — I'm not saying this absolutely should be used with qemu-user; I've had a bit of trouble with qemu-user too. My primary motivation was to allow ARM32-capable aarch64 machines to build ARM32 stuff. Another option for this would of course be to have both 32-bit and 64-bit nix installed on the machine, but at that point working out which to use, especially in automated invocations, could get hairy.

shlevy · 2018-02-28T17:21:32Z

Oh, yeah, that's a solid use case 👍

cleverca22 · 2018-03-16T22:37:02Z

another edgecase this can fix, some of the kernel options in nixos lack 32bit x86 support, and there is currently no way to tell nix about that

but if the automatic 64bit supports 32bit too code is removed, we can declare the 32bit support in the config file

also document it

lheckemann · 2018-03-16T22:52:42Z

Added some documentation, renamed the option, added i686-linux on x86_64-linux by default. I'm not 100% happy with the description, advice on improving it would be appreciated. I'll also squash the commits before this gets merged.

Support extra compatible architectures (#1916)

edolstra · 2018-05-03T11:32:28Z

I don't really see the use case for this option. Nix needs to know how to build something for a particular platform type. For example, on x86_64-linux, we can support i686-linux because there is a personality() syscall to do so. Merely adding a platform to an extra-platforms option isn't enough.

edolstra · 2018-05-03T11:34:27Z

doc/manual/command-ref/conf-file.xml

+    misdetect the target platform and generate incompatible code, so you
+    may wish to cross-check the results of using this option against
+    proper natively-built versions of your
+    derivations.</para></listitem>


This warning suggests that extra-platforms is a really bad idea. I mean, it allows you to completely lie to Nix about the platform type (e.g. do a x86_64-darwin build on x86_64-linux), thus producing a completely wrong build result, which could then end up in a binary cache.

if you lie and dont configure things properly, it will simply fail to execute the darwin bash/gcc/bootstrap tools, and never get anywhere

lheckemann · 2018-05-03T11:35:22Z

I've tested this and it works on an aarch64 machine with 32-bit support. Nixpkgs does need quite a few modifications for this to work (NixOS/nixpkgs#21471 should fix this properly) correctly, but I have got it producing binaries that work fine on a 32-bit only ARM machine successfully.

edolstra · 2018-05-03T11:39:28Z

Why does Nixpkgs need modification?

lheckemann · 2018-05-03T11:41:12Z

localhost:/tmp# nix build -f '<nixpkgs>' --argstr system armv7l-linux hello
[1 built, 1 copied (0.7 MiB), 0.0 MiB DL]
localhost:/tmp# uname -a
Linux localhost 4.9.20 #1 SMP Tue Sep 12 19:34:09 UTC 2017 aarch64 GNU/Linux
localhost:/tmp# file result/
bin/    share/
localhost:/tmp# file result/bin/hello
result/bin/hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/lhsrzki5bx9bj2ccxibxw5scfckh3p01-glibc-2.27/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped
localhost:/tmp# result/bin/hello
Hello, world!

The reason nixpkgs needs modification is that a lot of packages will incorrectly detect the system (using uname AFAIK) to build for as aarch64 when we actually want to build for armv7 using armv7 tools.

edolstra · 2018-05-03T11:47:22Z

Right, that demonstrates exactly why this option is a bad idea. You could do a armv7 build and end up with a aarch64 binary, a rather significant impurity. For x86_64-linux/i686-linux we have personality() to prevent that. Maybe aarch64 has something similar?

IMHO this PR should be reverted. Or at the very least extra-platforms should be undocumented.

lheckemann · 2018-05-03T11:49:04Z

Maybe aarch64 has something similar?

It doesn't, as far as I know.

You could do a armv7 build and end up with a aarch64 binary, a rather significant impurity.

Exactly the same thing can still be achieved by using an armv7 nix on the same machine, so the absence of this option doesn't eliminate this impurity risk.

edolstra · 2018-05-03T11:57:56Z

It doesn't, as far as I know.

Maybe seccomp can be used to fake the uname result?

Exactly the same thing can still be achieved by using an armv7 nix on the same machine, so the absence of this option doesn't eliminate this impurity risk.

Yes, there is the same problem when running i686-linux on x86_64-linux since then it doesn't know that it should use personality(). I guess we should move the build-time determination of the platform to runtime.

lheckemann · 2018-05-03T12:07:23Z

Maybe seccomp can be used to fake the uname result?

Sure, if that works!

Yes, there is the same problem when running i686-linux on x86_64-linux since then it doesn't know that it should use personality(). I guess we should move the build-time determination of the platform to runtime.

Yes, but I'm also strongly in favour of having nixpkgs tell the build systems what to build for explicitly as per NixOS/nixpkgs#21471.

What I don't want is to end up having machines (probably more powerful than any "pure" 32-bit ARM machine) that can run a 32-bit user space but not allowing nix to build 32-bit arm stuff on them, just because they also support aarch64.

Ericson2314 · 2018-05-04T18:18:23Z

Maybe seccomp can be used to fake the uname result?

That + NixOS/nixpkgs#21471 seems like a good thing.

dezgeg · 2018-05-08T21:05:11Z

There is already some discussion on the uname thing at NixOS/nixpkgs#25240. So, there's no personality() flag that is good enough. And seccomp (except until this SECCOMP_RET_USER_NOTIF someone proposed on LKML goes in) would mean using ptrace(), which is pretty much a no-go. I also don't expect NixOS/nixpkgs#21471 to ever happen (if it means that configure scripts will start spewing those configure: error: cannot run test program while cross compiling. things at users building native stuff).

What to me sounds like the best next plan is to try exporting build_alias to the host triplet in native builds, which should make autotools not call uname without hopefully breaking anything else. Sadly, I think some of the known offenders (on the ARMv6-on-ARMv7 case at least) like perl and openssl don't use autotools...

dezgeg · 2018-05-08T21:12:15Z

BTW so far I've found out it's easiest to boot a ARMv7 KVM virtual machine on aarch64 to sidestep this problem :-)

Ericson2314 · 2018-05-08T21:23:18Z

if it means...

I don't expect to disable all of those and live either, but I'd like to do the rest of that issue.

dezgeg · 2018-05-09T12:39:01Z

(Getting a bit off-topic, but...) Well, the condition for that spew is literally:

AS_IF([test "$cross_compiling" = yes],
  [m4_default([$4],
	   [AC_MSG_FAILURE([cannot run test program while cross compiling])])],
  [_AC_RUN_IFELSE($@)])
])

so there is no way to simultaneously have autotools in cross-compilation mode and have those compile+run tests getting executed.

dezgeg · 2018-05-09T14:37:48Z

I tried the quick hack of doing a export build_alias='armv6l-unknown-linux-gnueabihf' in stdenv and the results were quite good; Ruby created a lib/ruby/vendor_ruby/2.4.0/armv6l-linux-eabihf as expected and GMP stopped looking in /proc/cpuinfo and enabling NEON instructions (IIRC GMP currently has an impurity problem on x86 as well where it does similar things). Perl and OpenSSL didn't get fixed as expected (since they don't use autotools).

lopsided98 · 2018-05-20T16:25:39Z

I built an armv6l system on armv7l, and it seems to be working well. So far, I've only had to modify openssl and I haven't run into any other illegal instructions yet.

@dezgeg Where did you set build_alias? In mkDerivation, or somewhere else? I was thinking about setting configurePlatforms to [ "build" ] by default, but this fails because it passes --build to all configure scripts, which often fails if the package does not use autotools. Should configurePlatforms set build/host/target_alias instead?

dezgeg · 2018-05-20T16:31:05Z

See NixOS/nixpkgs@master...dezgeg:uname-hack.

dezgeg · 2018-06-04T12:48:32Z

Seems like this SECCOMP_RET_USER_NOTIF is gaining some traction (https://lwn.net/SubscriberLink/756233/1f23df75d146c393/) so hopefully we get to use it for uname() emulation.

In the meantime I've experimented with a dirty kernel hack to look at the system environment variable and base the uname -m result on that: https://github.com/dezgeg/nixpkgs/blob/arm-work/pkgs/os-specific/linux/kernel/uname-hack.patch

Don't use yet, it still has some problems with the huge environment variables that happen in nixpkgs stdenv.

lopsided98 · 2018-08-05T17:50:21Z

I've been building for armv6l on armv7l using build_alias and a fix for openssl to prevent it from detecting the platform. I'm sure there are packages that still have problems, but it has been working well for my use cases. My armv6l machines are so slow that this is the only way I can feasibly run NixOS on them, so I would like to see this configuration be better supported in nixpkgs.

lheckemann added 2 commits February 27, 2018 00:27

shlevy reviewed Feb 27, 2018

View reviewed changes

lheckemann added 2 commits March 16, 2018 22:50

rename build-extra-platforms -> extra-platforms

637701b

also document it

build-remote: take extra-platforms into account

639c166

shlevy approved these changes Mar 17, 2018

View reviewed changes

shlevy added the backlog label Apr 1, 2018

shlevy self-assigned this Apr 1, 2018

shlevy merged commit 639c166 into NixOS:master Apr 23, 2018

shlevy added a commit that referenced this pull request Apr 23, 2018

Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix

8e6108f

Support extra compatible architectures (#1916)

lheckemann deleted the aarch64-armv7 branch April 23, 2018 13:03

edolstra reviewed May 3, 2018

View reviewed changes

lopsided98 mentioned this pull request Oct 6, 2018

openssl: don't autodetect platform on armv6/7l NixOS/nixpkgs#47953

Merged

9 tasks

matthewbauer mentioned this pull request Dec 1, 2018

binfmt_misc support for cross compiling NixOS/nixpkgs#33553

Closed

expipiplus1 mentioned this pull request Aug 19, 2019

Add a fake uname implementation to the Linux stdenv (that returns hardcoded values) NixOS/nixpkgs#25240

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support extra compatible architectures #1916

Support extra compatible architectures #1916

lheckemann commented Feb 27, 2018

shlevy left a comment

shlevy Feb 27, 2018

lheckemann Feb 28, 2018

Ericson2314 Mar 1, 2018 •

edited

lheckemann Mar 16, 2018

bjornfor commented Feb 27, 2018

lheckemann commented Feb 28, 2018 •

edited

shlevy commented Feb 28, 2018

cleverca22 commented Mar 16, 2018

lheckemann commented Mar 16, 2018

edolstra commented May 3, 2018

edolstra May 3, 2018

cleverca22 May 8, 2018

lheckemann commented May 3, 2018

edolstra commented May 3, 2018

lheckemann commented May 3, 2018

edolstra commented May 3, 2018

lheckemann commented May 3, 2018

edolstra commented May 3, 2018

lheckemann commented May 3, 2018

Ericson2314 commented May 4, 2018

dezgeg commented May 8, 2018

dezgeg commented May 8, 2018

Ericson2314 commented May 8, 2018 •

edited

dezgeg commented May 9, 2018 •

edited

dezgeg commented May 9, 2018

lopsided98 commented May 20, 2018

dezgeg commented May 20, 2018

dezgeg commented Jun 4, 2018

lopsided98 commented Aug 5, 2018

Support extra compatible architectures #1916

Support extra compatible architectures #1916

Conversation

lheckemann commented Feb 27, 2018

shlevy left a comment

Choose a reason for hiding this comment

shlevy Feb 27, 2018

Choose a reason for hiding this comment

lheckemann Feb 28, 2018

Choose a reason for hiding this comment

Ericson2314 Mar 1, 2018 • edited

Choose a reason for hiding this comment

lheckemann Mar 16, 2018

Choose a reason for hiding this comment

bjornfor commented Feb 27, 2018

lheckemann commented Feb 28, 2018 • edited

shlevy commented Feb 28, 2018

cleverca22 commented Mar 16, 2018

lheckemann commented Mar 16, 2018

edolstra commented May 3, 2018

edolstra May 3, 2018

Choose a reason for hiding this comment

cleverca22 May 8, 2018

Choose a reason for hiding this comment

lheckemann commented May 3, 2018

edolstra commented May 3, 2018

lheckemann commented May 3, 2018

edolstra commented May 3, 2018

lheckemann commented May 3, 2018

edolstra commented May 3, 2018

lheckemann commented May 3, 2018

Ericson2314 commented May 4, 2018

dezgeg commented May 8, 2018

dezgeg commented May 8, 2018

Ericson2314 commented May 8, 2018 • edited

dezgeg commented May 9, 2018 • edited

dezgeg commented May 9, 2018

lopsided98 commented May 20, 2018

dezgeg commented May 20, 2018

dezgeg commented Jun 4, 2018

lopsided98 commented Aug 5, 2018

Ericson2314 Mar 1, 2018 •

edited

lheckemann commented Feb 28, 2018 •

edited

Ericson2314 commented May 8, 2018 •

edited

dezgeg commented May 9, 2018 •

edited