emacs: Make gccemacs build on darwin #94637

antifuchs · 2020-08-04T03:47:34Z

Motivation for this change

To allow macOS/darwin users to build Emacs with the native-comp feature. These changes are all for problems that on Darwin, result in broken binary installs and failures to resolve the emacs derivation.

Things done

antifuchs · 2020-08-04T03:48:32Z

@evanrelf I decided to open a real PR & take this here from #93716. Am currently building libgccjit from this branch and I suspect that this time, the emacs configure stage won't error out... but it'll be 2 hours /:

evanrelf · 2020-08-04T05:08:37Z

@antifuchs Awesome! And thanks for letting me know 🙂

antifuchs · 2020-08-04T16:25:18Z

OK, I finally got this to build properly, and it looks like the native-comp emacs branch is working on this Darwin installation:

native GUI
compilation at runtime
json via libjason

antifuchs · 2020-08-04T17:52:09Z

One more thing that is currently failing:

/nix/store/6s7nbin6khpmc6f6nwvcxl3m3h48c3fn-binutils-wrapper-2.31.1/bin/ld: line 207: /nix/store/7gmqwn05rw84kx06wyqhg7ciyn34j0qa-binutils-2.31.1/bin/ld: No such file or directory - seems like binutils aren't present, preventing native-compile from working properly (it compiles to .elc regardless)

antifuchs · 2020-08-05T01:22:47Z

So this looks like the default native-compilation $PATH that gets set uses the binutils-wrapper ld, which exists, but the wrapped ld doesn't: there's actually a comment about it in the darwin binutils wrappers. I'm not sure what to do about this, or where this comes from (and why AOT compilation during build works, it produces .eln files then!)

Mic92 · 2020-08-05T06:12:50Z

pkgs/development/compilers/gcc/builder.sh

@@ -251,7 +251,7 @@ postInstall() {
    fi

    if type "install_name_tool"; then
-        for i in "${!outputLib}"/lib/*.*.dylib; do
+        for i in "${!outputLib}"/lib/*.*.dylib "${!outputLib}"/lib/*.so.[0-9]; do


Is it a bug in gcc buildsystem that they install libgccjit as .so file?

Looks like this needs to be fixed here: https://github.com/gcc-mirror/gcc/blob/41d6b10e96a1de98e90a7c0378437c3255814b16/gcc/jit/Make-lang.in#L43

Yep, looks like it. They still make weird linux-soname based assumptions around how the structure works, but I think we could just set this as a make variable (or patch it) to fix. Probably the better way to fix it!

They have it conditionals for windows but not for macOS now: https://github.com/gcc-mirror/gcc/blob/229752afe3156a3990dacaedb94c76846cebf132/gcc/jit/Make-lang.in#L48

Anyway if you don't patch the build system would it not be better to rename it from .so to .dylib?

antifuchs · 2020-08-08T14:44:40Z

So as far as I can tell, on Darwin you must pass linker options (especially the ones used to find libgccjit) to ld via command-line args, and the LIBRARY_PATH env variable does nothing, even with wrapper scripts. That complicates post-install AOT compilation a bit. I filed an enhancement request with the gnu emacs tracker, to let users set driver options, but until then I don't think this can work... unless we build & ship a custom wrapper script.

antifuchs · 2020-08-24T13:48:39Z

Status update on this:

I have to fix merge conflicts on this.
There's a change pending that will add the ability to set linker paths directly via an elisp variable, in https://github.com/emacs-mirror/emacs/tree/scratch/native-comp-gcc-driver-options (I'm building emacs against my changes).
With that change, I have an outstanding change to make use of that facility, which allows native-comp emacs to run and compile elisp on Darwin.
I also tried actioning @Mic92's suggestion of patching the libgccjit build process, but could not figure out a way yet to make it properly install the versioned dylib file and a symlink to it... something always clobbers that final symlink. As re-building libgccjit takes 3-4 hours each go-around and the problem occurs at the final install phase, I'd prefer to stick to installing the .so for now (as it works regardless of file name), and fix that build process in a follow-up PR.

I'll push changes to make this happen as soon as the scratch branch is on the native-comp feature branch, hopefully the next couple of days!

The default `strip` invocation tries to strip global symbols from the library, and refuses because those are indirect symbol table references.

This changes PATH such that the correct linker can be found to construct .eln files at runtime.

Since Darwin's linker does not understand LIBRARY_PATH, we have to set the library path as explicit linker flags: This requires a very recent feature/native-comp emacs revision, but it runs on Darwin and correctly compiles files at runtime.

It's not necessary to use strip -x otherwise, so let's just use it for the JIT library.

This is the portable way to address the system-appropriate libc, better than conditionalizing by (darwin/GNU) system or using either one.

antifuchs · 2020-08-30T03:13:43Z

I'm not able to test this branch locally on linux as p11-kit tests fail while building emacs. This isn't happening on master for me.

That's curious - I could build this derivation just fine on Ubuntu 18.04. ~~I am, however, getting a weird test failure in something else that this PR doesn't touch /:~~ (Update: That was a timing problem in the python pytest suite; installing that derivation in single-core mode got me a functional emacs on my Linux box).

pkgs/applications/editors/emacs/generic.nix

This way, we don't have to drag clang or binutils/binutils-wrapped into the emacs closure, and can instead rely on using the correct one for the platform we're running on. Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>

The -B flag to gcc (and libgccjit) allows us to specify where it can find things it needs to correctly compile code (both programs and libraries) without adjusting any environmental flags: So, no need to wrap the program for a PATH entry containing binutils, and no need to explicitly pass a linker path anymore.

pkgs/applications/editors/emacs/generic.nix

The -B flag to gcc (and libgccjit) allows us to specify where it can find things it needs to correctly compile code (both programs and libraries) without adjusting any environmental flags: So, no need to wrap the program for a PATH entry containing binutils, and no need to explicitly pass a linker path anymore.

rcorrear · 2020-09-04T11:23:56Z

Hi all! Been following the discussion lately, haven't tried it but thanks for all the effort! A question comes to mind, has anyone tried the patch for "emacsMac" here?

mjlbach · 2020-09-04T14:56:38Z

@rcorrear If you mean, will native-compilation (gccemacs) work with the Mitsuharu's emacs-mac repository, the answer is no. Native compilation is currently in a feature branch, and it would be nontrivial to merge mitsuharu's work. You'll likely have to wait until native compilation is merged into emacs.

rcorrear · 2020-09-04T16:30:18Z

@mjlbach yeah, that's what I wanted to know, thanks!

adelbertc · 2020-09-20T02:40:42Z

Apologies if this is an inappropriate place to ask, but does anyone know if there is a cached build of emacsGcc on macOS available? I just tried building it with something like:

    emacsPackagesNg =
      let
        emacs-overlay =
          import (builtins.fetchTarball https://github.com/nix-community/emacs-overlay/archive/master.tar.gz);

        pkgs = import (fetchGit {
          url = "https://github.com/NixOS/nixpkgs.git";
          rev = "6fc3562432357cdbcbb30804b47feacfac452acb"; # this is this PR's merge commit
        }) {
          overlays = [ emacs-overlay ];
        };
      in
        pkgs.emacsPackagesNgGen pkgs.emacsGcc;

on a 2017 13" MacBook Pro and it was compiling from like 11:30AM - 7:30PM before I Ctrl+C'd it, and even after all that time it was still building LLVM things 😬

mjlbach · 2020-09-20T03:27:46Z

If you want, you can use my overlay/cachix which has CI for macOS (only on the linked branch).

https://github.com/mjlbach/emacs-pgtk-nativecomp-overlay/tree/cleanups

nixos-discourse · 2020-09-25T05:54:13Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/lsopenurlswithrole-error-opening-nix-installed-application-emacs-native-comp-on-macos/9132/1

antifuchs requested review from adisbladis and matthewbauer as code owners August 4, 2020 03:47

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Aug 4, 2020

antifuchs changed the title ~~[wip] Make gccemacs build on darwin~~ [wip] emacs: Make gccemacs build on darwin Aug 4, 2020

ofborg bot added the 6.topic: emacs label Aug 4, 2020

ofborg bot added the 10.rebuild-linux-stdenv label Aug 4, 2020

ofborg bot requested review from jwiegley, lovek323 and peti August 4, 2020 05:53

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Aug 4, 2020

antifuchs changed the title ~~[wip] emacs: Make gccemacs build on darwin~~ emacs: Make gccemacs build on darwin Aug 4, 2020

antifuchs force-pushed the gccemacs-on-darwin branch from 9b1eb34 to 1083b43 Compare August 5, 2020 00:50

Mic92 requested a review from LnL7 August 5, 2020 06:10

Mic92 reviewed Aug 5, 2020

View reviewed changes

antifuchs force-pushed the gccemacs-on-darwin branch from 1083b43 to 8620997 Compare August 5, 2020 14:09

jwiegley approved these changes Aug 5, 2020

View reviewed changes

FRidh added this to WIP in Staging via automation Aug 15, 2020

Staging automation moved this from WIP to Ready Aug 17, 2020

matthewbauer approved these changes Aug 17, 2020

View reviewed changes

gcc: Fix building libgccjit on darwin, using strip -x

038e86f

The default `strip` invocation tries to strip global symbols from the library, and refuses because those are indirect symbol table references.

mjlbach mentioned this pull request Aug 29, 2020

emacsGcc deffered compliation fails nix-community/emacs-overlay#33

Closed

This comment has been minimized.

Sign in to view

antifuchs added 4 commits August 29, 2020 23:10

emacs: Fix paths to bintools on Darwin

861f270

This changes PATH such that the correct linker can be found to construct .eln files at runtime.

gcc: Only use strip -x if building libgccjit on darwin

41e34e7

It's not necessary to use strip -x otherwise, so let's just use it for the JIT library.

emacs: Use stdenv.cc.libc instead of plain .libc

ad7a3fb

This is the portable way to address the system-appropriate libc, better than conditionalizing by (darwin/GNU) system or using either one.

antifuchs force-pushed the gccemacs-on-darwin branch from 4334b63 to ad7a3fb Compare August 30, 2020 03:11

matthewbauer reviewed Aug 30, 2020

View reviewed changes

pkgs/applications/editors/emacs/generic.nix Outdated Show resolved Hide resolved

ofborg bot removed the 10.rebuild-darwin-stdenv label Aug 30, 2020

emacs: Use stdenv's bintools instead of clang

3384837

This way, we don't have to drag clang or binutils/binutils-wrapped into the emacs closure, and can instead rely on using the correct one for the platform we're running on. Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>

mjlbach mentioned this pull request Aug 31, 2020

p11-kit fails to build locally on non-NixOS systems (single-user install) due to failing tests on master #96715

Open

mjlbach mentioned this pull request Aug 31, 2020

Emacs: Fix library_path substitution to not be a nested list for native compilation #94154

Closed

10 tasks

mjlbach reviewed Sep 1, 2020

View reviewed changes

pkgs/applications/editors/emacs/generic.nix Outdated Show resolved Hide resolved

antifuchs force-pushed the gccemacs-on-darwin branch from 43baf60 to 04fffd6 Compare September 2, 2020 02:58

matthewbauer approved these changes Sep 2, 2020

View reviewed changes

adisbladis merged commit 6fc3562 into NixOS:staging Sep 2, 2020

Staging automation moved this from Ready to Done Sep 2, 2020

antifuchs deleted the gccemacs-on-darwin branch September 2, 2020 22:20

adisbladis mentioned this pull request Sep 23, 2020

EmacsWithPackages' wrapper.nix should take an arguments parameter for emacs byte compilation #90169

Closed

collares mentioned this pull request Feb 26, 2021

libgccjit: use current gcc version #111166

Merged

3 tasks

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

emacs: Make gccemacs build on darwin #94637

emacs: Make gccemacs build on darwin #94637

antifuchs commented Aug 4, 2020 •

edited

antifuchs commented Aug 4, 2020

evanrelf commented Aug 4, 2020

antifuchs commented Aug 4, 2020 •

edited

antifuchs commented Aug 4, 2020

antifuchs commented Aug 5, 2020

Mic92 Aug 5, 2020

Mic92 Aug 5, 2020

antifuchs Aug 5, 2020

Mic92 Aug 5, 2020

Mic92 Aug 5, 2020 •

edited

antifuchs commented Aug 8, 2020 •

edited

antifuchs commented Aug 24, 2020 •

edited

This comment has been minimized.

antifuchs commented Aug 30, 2020 •

edited

rcorrear commented Sep 4, 2020

mjlbach commented Sep 4, 2020

rcorrear commented Sep 4, 2020

adelbertc commented Sep 20, 2020

mjlbach commented Sep 20, 2020 •

edited

nixos-discourse commented Sep 25, 2020

emacs: Make gccemacs build on darwin #94637

emacs: Make gccemacs build on darwin #94637

Conversation

antifuchs commented Aug 4, 2020 • edited

Motivation for this change

Things done

antifuchs commented Aug 4, 2020

evanrelf commented Aug 4, 2020

antifuchs commented Aug 4, 2020 • edited

antifuchs commented Aug 4, 2020

antifuchs commented Aug 5, 2020

Mic92 Aug 5, 2020

Choose a reason for hiding this comment

Mic92 Aug 5, 2020

Choose a reason for hiding this comment

antifuchs Aug 5, 2020

Choose a reason for hiding this comment

Mic92 Aug 5, 2020

Choose a reason for hiding this comment

Mic92 Aug 5, 2020 • edited

Choose a reason for hiding this comment

antifuchs commented Aug 8, 2020 • edited

antifuchs commented Aug 24, 2020 • edited

This comment has been minimized.

antifuchs commented Aug 30, 2020 • edited

rcorrear commented Sep 4, 2020

mjlbach commented Sep 4, 2020

rcorrear commented Sep 4, 2020

adelbertc commented Sep 20, 2020

mjlbach commented Sep 20, 2020 • edited

nixos-discourse commented Sep 25, 2020

antifuchs commented Aug 4, 2020 •

edited

antifuchs commented Aug 4, 2020 •

edited

Mic92 Aug 5, 2020 •

edited

antifuchs commented Aug 8, 2020 •

edited

antifuchs commented Aug 24, 2020 •

edited

antifuchs commented Aug 30, 2020 •

edited

mjlbach commented Sep 20, 2020 •

edited