Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add x86_64-darwin and aarch64 to "extra-platforms" automatically when Rosetta2 is detected #4310

Merged
merged 3 commits into from Jan 6, 2021

Conversation

matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Dec 3, 2020

First:

Add extraPlatforms for Rosetta 2 macOS

macOS systems with ARM64 can utilize a translation layer at /Library/Apple/usr/libexec/oah to run x86_64 binaries. This change makes Nix recognize that and it to "extra-platforms". Note that there are two cases here since Nix could be built for either x86_64 or aarch64. In either case, we can switch to the other architecture. Unfortunately there is not a good way to prevent aarch64 binaries from being run in x86_64 contexts or vice versa - programs can always execute programs for the other architecture.

Question: Can someone confirm that /Library/Apple/usr/libexec/oah does not exist when Rosetta 2 is not installed? There could be some stubs there even if Rosetta 2 is not setup yet.

Second:

Use posix_spawn_setbinpref_np to advise which architecture to run

When running universal binaries like /bin/bash, Darwin XNU will choose which architecture of the binary to use based on "binary preferences". This change sets that to the current platform for aarch64 and x86_64 builds. In addition it now uses posix_spawn instead of the usual execve. Note, that this does not prevent the other architecture from being run, just advises which to use.

/cc @abathur @thefloweringash @edolstra

macOS systems with ARM64 can utilize a translation layer at
/Library/Apple/usr/libexec/oah to run x86_64 binaries. This change
makes Nix recognize that and it to "extra-platforms". Note that there
are two cases here since Nix could be built for either x86_64 or
aarch64. In either case, we can switch to the other architecture.
Unfortunately there is not a good way to prevent aarch64 binaries from
being run in x86_64 contexts or vice versa - programs can always
execute programs for the other architecture.
When running universal binaries like /bin/bash, Darwin XNU will choose
which architecture of the binary to use based on "binary preferences".
This change sets that to the current platform for aarch64 and x86_64
builds. In addition it now uses posix_spawn instead of the usual
execve. Note, that this does not prevent the other architecture from
being run, just advises which to use.

Unfortunately, posix_spawnattr_setbinpref_np does not appear to be
inherited by child processes in x86_64 Rosetta 2 translations, meaning
that this will not always work as expected.

For example:

  {
    arm = derivation {
      name = "test";
      system = "aarch64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
    rosetta = derivation {
      name = "test";
      system = "x86_64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = i386 ]
        echo It works!
        /usr/bin/touch $out
      '') ];
    };
  }

`arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on
aarch64-compiled Nix. I suspect there is a way to fix this since:

  $ /usr/bin/arch -arch x86_64 /bin/bash \
    -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch'
  arm64

seems to work correctly. We may need to wait for Apple to update
system_cmds in opensource.apple.com to find out how though.
@matthewbauer matthewbauer changed the title Add "extra-platforms" automatically for Add x86_64-darwin and aarch64 to "extra-platforms" automatically when Rosetta2 is detected Dec 3, 2020
@abathur
Copy link
Member

abathur commented Dec 4, 2020

Not sure which bits you're hoping I'll have feedback on; @LnL7 is almost certainly a better mention if he has bandwidth!

It's not clear to me exactly what you mean or under what circumstances Rosetta 2 isn't installed or set up (perhaps it would be obvious if I had access to an ARM mac?), but I do still have an intel Big Sur install, so I did run find / -name oah to avoid being useless:

  • Catalina: no results
  • Intel Big Sur: (I actually see 6 different paths, but I think they're all either the libexec or db version of the path)
    • /System/Library/ProxySigningStubsLibrary/Apple/usr/libexec/oah (file says: directory)
      • This has files: debugserver, oahd, oahd-helper, oahd-root-helper, runtime, runtime_t8027, translate_tool
    • /System/Library/Templates/Data/private/var/db/oah (file says: directory)
      • ls says no perm, I assume it needs full disk

@thefloweringash
Copy link
Member

I see an unofficial install script for Rosetta 2 that uses the presence of /Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist to detect if it has been installed. On my arm64 mac with Rosetta installed I can confirm this file exists.

@abathur
Copy link
Member

abathur commented Dec 4, 2020

sudo find / -name com.apple.oahd.plist came up empty on both catalina and intel big sur

@abathur
Copy link
Member

abathur commented Dec 4, 2020

Also, /usr/sbin/softwareupdate does list the --install-rosetta option (Install Rosetta 2), and running it on Intel yields Installing Rosetta 2 on this system is not supported. (exit status 1)

@Kloenk
Copy link
Member

Kloenk commented Dec 4, 2020

I see an unofficial install script for Rosetta 2 that uses the presence of /Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist to detect if it has been installed. On my arm64 mac with Rosetta installed I can confirm this file exists.

Hopefully around Tuesday I can look if that path exists, before installing Rosetta 2 if that helps.

@Kloenk
Copy link
Member

Kloenk commented Dec 17, 2020

What's the state of this one? The path was correct. (Only forgot to confirm it here though)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/macos-m1-issues/10771/5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants