Even more android fixes #42412

matthewbauer · 2018-06-23T02:00:32Z

Motivation for this change

This is a collection of tweaks for handling android. dc261f6896106deab2efd6329479d536d3e9c255 is the commit that solves my original problem & definitely should be merged if possible as it isn't a mass rebuild.

On the other hand, 62024d280eb7ecd1f49d80406193a65901317bcb, may not be that useful but it seems like a safe thing to just disable stripping & patchelf in cross drvs. Interested in feedback on whether this is a good idea.

It doesn't seem like 25e54c53a25f673f2ada852ae143f17dd1280842 should be needed but it fixed a non-fatal warning I was getting & might give us trouble if we don't properly set up the rpath.

GrahamcOfBorg · 2018-06-23T02:22:45Z

Success on aarch64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/jcs8jiz94na13d960b8hd8b05j58i80x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T02:22:47Z

Success on x86_64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/bbg8p1v9qhjfr0ghlzi8bn3f23wlj4rx-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T02:22:59Z

Success on x86_64-darwin (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/znlnr5p1n53n07d2mci10d2byl43gl2p-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-23T04:50:09Z

Success on x86_64-darwin (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/znlnr5p1n53n07d2mci10d2byl43gl2p-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-23T04:50:22Z

Success on aarch64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

  /nix/store/5fpq3zhq8dmvvc10q62i797j88l655y1-linux-headers-4.15
  /nix/store/csqjc2aycrin72amhc1xn49wwmcijddc-glibc-2.27-dev
  /nix/store/d5djbvna6ffvviqnkpy29p0798zprchr-glibc-2.27
  /nix/store/jcs8jiz94na13d960b8hd8b05j58i80x-glibc-iconv-2.27
copying path '/nix/store/d5djbvna6ffvviqnkpy29p0798zprchr-glibc-2.27' from 'https://cache.nixos.org'...
copying path '/nix/store/5fpq3zhq8dmvvc10q62i797j88l655y1-linux-headers-4.15' from 'https://cache.nixos.org'...
copying path '/nix/store/3nmzj02hcf38zzg1s6cl63g7n3801l5x-glibc-2.27-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/csqjc2aycrin72amhc1xn49wwmcijddc-glibc-2.27-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/jcs8jiz94na13d960b8hd8b05j58i80x-glibc-iconv-2.27' from 'https://cache.nixos.org'...
/nix/store/jcs8jiz94na13d960b8hd8b05j58i80x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T04:50:25Z

Success on x86_64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/bbg8p1v9qhjfr0ghlzi8bn3f23wlj4rx-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T05:09:12Z

Success on aarch64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/jcs8jiz94na13d960b8hd8b05j58i80x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T05:09:18Z

Success on x86_64-darwin (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/znlnr5p1n53n07d2mci10d2byl43gl2p-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-23T05:09:26Z

Success on x86_64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/bbg8p1v9qhjfr0ghlzi8bn3f23wlj4rx-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T05:29:23Z

Success on x86_64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/bbg8p1v9qhjfr0ghlzi8bn3f23wlj4rx-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T05:38:23Z

Success on x86_64-darwin (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/znlnr5p1n53n07d2mci10d2byl43gl2p-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-23T05:38:24Z

Success on x86_64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/bbg8p1v9qhjfr0ghlzi8bn3f23wlj4rx-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T05:38:41Z

Success on aarch64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/jcs8jiz94na13d960b8hd8b05j58i80x-glibc-iconv-2.27

matthewbauer · 2018-06-23T16:03:27Z

pkgs/development/mobile/androidenv/androidndk-pkgs.nix

@@ -84,6 +84,7 @@ rec {
          $out/bin/${targetPlatform.config}-g++ \
          -e '130i    extraBefore+=(-Wl,--fix-cortex-a8)' \
          -e 's|^(extraBefore=)\(\)$|\1(${builtins.toString flags})|'
+        echo "-no-pie" >> $out/nix-support/cc-cflags


/cc @Ericson2314 Is this okay to do globally?

Yeah sounds good. I'd maybe put it in the extraBefore thing above, but fine either way.

Ericson2314 · 2018-06-23T16:37:56Z

pkgs/os-specific/darwin/apple-source-releases/libiconv/default.nix

-{ stdenv, appleDerivation, autoreconfHook, targetPlatform, enableStatic ? targetPlatform.isiOS }:
+{ stdenv, appleDerivation, lib, autoreconfHook, targetPlatform
+, enableStatic ? targetPlatform.isiOS
+, enableShared ? !targetPlatform.isiOS }:


can the }: be on its own line?

Ericson2314 · 2018-06-23T16:38:57Z

pkgs/os-specific/darwin/apple-source-releases/libiconv/default.nix

    sed -i 's/darwin\*/ios\*/g' configure libcharset/configure
  '';

-  configureFlags = stdenv.lib.optionals enableStatic [ "--enable-static" "--disable-shared" ];
+  configureFlags = lib.optional enableStatic "--enable-static"


let's use https://github.com/NixOS/nixpkgs/blob/master/lib/strings.nix#L415 on staging after. Sound good?

Sounds good

(And same for normal libiconv)

matthewbauer · 2018-06-23T17:52:45Z

Waiting on: https://hydra.reflex-frp.org/build/685

GrahamcOfBorg · 2018-06-23T18:40:36Z

Success on x86_64-linux (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-23T18:40:42Z

Success on x86_64-darwin (full log)

Attempted: libiconv

Partial log (click to expand)

/nix/store/y0b2y33wzmlijsq81x831k3cy00v2vx4-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-25T14:00:49Z

Success on x86_64-linux (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/i2d6l6w99qlwhz6gbz4hxl4d9238drfj-ghc-8.2.2
/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

Ericson2314 · 2018-06-25T15:33:35Z

pkgs/top-level/impure.nix

@@ -83,5 +83,6 @@ import ./. (builtins.removeAttrs args [ "system" "platform" ] // {
  inherit config overlays crossSystem;
  # Fallback: Assume we are building packages on the current (build, in GNU
  # Autotools parlance) system.
-  localSystem = { system = builtins.currentSystem; } // localSystem;
+  localSystem = (if args ? localSystem then {}
+                 else { system = builtins.currentSystem; }) // localSystem;


Ah, nice catch.

GrahamcOfBorg · 2018-06-25T18:00:18Z

Success on x86_64-linux (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/i2d6l6w99qlwhz6gbz4hxl4d9238drfj-ghc-8.2.2
/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T18:00:25Z

Success on aarch64-linux (full log)

Attempted: libiconv

The following builds were skipped because they don't evaluate on aarch64-linux: ghc

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/yhv02gycgbl4zrvpa8kjgxl7b1wh8c2x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T18:25:23Z

Success on x86_64-linux (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/i2d6l6w99qlwhz6gbz4hxl4d9238drfj-ghc-8.2.2
/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T18:25:28Z

Success on x86_64-darwin (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/bdvs7h7zirrfjrk7nn2cm2w8cnivyvdh-ghc-8.2.2
/nix/store/y0b2y33wzmlijsq81x831k3cy00v2vx4-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-25T18:26:27Z

Success on aarch64-linux (full log)

Attempted: libiconv

The following builds were skipped because they don't evaluate on aarch64-linux: ghc

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/yhv02gycgbl4zrvpa8kjgxl7b1wh8c2x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T19:08:57Z

Success on x86_64-linux (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/i2d6l6w99qlwhz6gbz4hxl4d9238drfj-ghc-8.2.2
/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T19:09:16Z

Success on aarch64-linux (full log)

Attempted: libiconv

The following builds were skipped because they don't evaluate on aarch64-linux: ghc

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/yhv02gycgbl4zrvpa8kjgxl7b1wh8c2x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T20:00:49Z

Success on aarch64-linux (full log)

Attempted: libiconv

The following builds were skipped because they don't evaluate on aarch64-linux: ghc

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/yhv02gycgbl4zrvpa8kjgxl7b1wh8c2x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T20:01:01Z

Success on x86_64-linux (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/i2d6l6w99qlwhz6gbz4hxl4d9238drfj-ghc-8.2.2
/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T20:34:28Z

Success on x86_64-darwin (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/bdvs7h7zirrfjrk7nn2cm2w8cnivyvdh-ghc-8.2.2
/nix/store/y0b2y33wzmlijsq81x831k3cy00v2vx4-libiconv-osx-10.11.6

When false, shared libraries will be disabled. also adds patch to *really* disable shared libiconv.

the gcc in android does not have this on by default. Doing this will fix some of the errors in cross-trunk: https://hydra.nixos.org/build/76354538/nixlog/11 This is related to how separate debug info is handled in Nixpkgs.

@bkchr

Setting this to true will disable removing some of the unused parts of the NDK toolchain that is unused in Nixpkgs but may be used by others. /cc @bkchr

We run into issues otherwise

This is needed for targeting older apis.

@Ericson2314

If we passed a localSystem in, we don’t want the current system to override it. Now we check for localSystem first to avoid getting "mixed" localSystem values from commands like this: nix-build --arg localSystem '{config="x86_64-unknown-linux-musl";}' -A hello Which would eventually evaluate localSystem to this: { config = "x86_64-unknown-linux-musl"; system = "x86_64-darwin"; } & Nix would not be able to run it correctly. Fixes NixOS#41599 /cc @Ericson2314

This breaks some library symbols apparently. ARM symbols must be incorrectly stripped. Also some very weird issues with patchelf.

GrahamcOfBorg · 2018-06-25T22:07:27Z

Success on x86_64-darwin (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/bdvs7h7zirrfjrk7nn2cm2w8cnivyvdh-ghc-8.2.2
/nix/store/y0b2y33wzmlijsq81x831k3cy00v2vx4-libiconv-osx-10.11.6

GrahamcOfBorg · 2018-06-25T22:07:47Z

Success on aarch64-linux (full log)

Attempted: libiconv

The following builds were skipped because they don't evaluate on aarch64-linux: ghc

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/yhv02gycgbl4zrvpa8kjgxl7b1wh8c2x-glibc-iconv-2.27

GrahamcOfBorg · 2018-06-25T22:07:56Z

Success on x86_64-linux (full log)

Attempted: ghc, libiconv

Partial log (click to expand)

/nix/store/i2d6l6w99qlwhz6gbz4hxl4d9238drfj-ghc-8.2.2
/nix/store/nbiacn62k38saz9mb108b13gp9zcmrzc-glibc-iconv-2.27

layus · 2018-06-26T09:37:27Z

Thanks! he "impure.nix: fix handling of localSystem" explains why it was such a pain for me to pass that argument.

GrahamcOfBorg added the 6.topic: stdenv Standard environment label Jun 23, 2018

matthewbauer requested a review from Ericson2314 June 23, 2018 02:09

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jun 23, 2018

matthewbauer force-pushed the even-more-android-fixes branch 2 times, most recently from 4dd5a98 to 92e8d15 Compare June 23, 2018 04:31

GrahamcOfBorg removed the 6.topic: stdenv Standard environment label Jun 23, 2018

matthewbauer force-pushed the even-more-android-fixes branch from 92e8d15 to 2399442 Compare June 23, 2018 04:55

matthewbauer force-pushed the even-more-android-fixes branch from 23e7aeb to 2d0b156 Compare June 23, 2018 05:24

matthewbauer commented Jun 23, 2018

View reviewed changes

Ericson2314 reviewed Jun 23, 2018

View reviewed changes

matthewbauer force-pushed the even-more-android-fixes branch from 2d0b156 to ff0c98d Compare June 23, 2018 17:45

Ericson2314 approved these changes Jun 23, 2018

View reviewed changes

Ericson2314 reviewed Jun 25, 2018

View reviewed changes

matthewbauer force-pushed the even-more-android-fixes branch from 0a086ba to 8425cc7 Compare June 25, 2018 17:46

matthewbauer force-pushed the even-more-android-fixes branch from 8425cc7 to 6188a0c Compare June 25, 2018 18:00

matthewbauer force-pushed the even-more-android-fixes branch from 6188a0c to 3e75f5e Compare June 25, 2018 18:38

matthewbauer force-pushed the even-more-android-fixes branch from 3e75f5e to e0e2af0 Compare June 25, 2018 19:45

matthewbauer added 7 commits June 25, 2018 17:10

libiconv: add "enableShared" option

73e2f60

When false, shared libraries will be disabled. also adds patch to *really* disable shared libiconv.

android: add --build-id to ldflags

7652b2e

the gcc in android does not have this on by default. Doing this will fix some of the errors in cross-trunk: https://hydra.nixos.org/build/76354538/nixlog/11 This is related to how separate debug info is handled in Nixpkgs.

androidndk: add fullNDK option

d915590

Setting this to true will disable removing some of the unused parts of the NDK toolchain that is unused in Nixpkgs but may be used by others. /cc @bkchr

ghc: build shared on android

2c2169c

We run into issues otherwise

android: define __ANDROID_API__ in cflags

5000cc5

This is needed for targeting older apis.

ghc843: don’t apply stripping in android

69cad91

This breaks some library symbols apparently. ARM symbols must be incorrectly stripped. Also some very weird issues with patchelf.

matthewbauer force-pushed the even-more-android-fixes branch from e0e2af0 to 69cad91 Compare June 25, 2018 21:13

matthewbauer merged commit 09c66f4 into NixOS:master Jun 25, 2018

Ericson2314 deleted the even-more-android-fixes branch June 26, 2018 17:21

Even more android fixes #42412

Even more android fixes #42412

Conversation

matthewbauer commented Jun 23, 2018 • edited by Ericson2314

Motivation for this change

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

matthewbauer Jun 23, 2018

Choose a reason for hiding this comment

Ericson2314 Jun 23, 2018

Choose a reason for hiding this comment

Ericson2314 Jun 23, 2018

Choose a reason for hiding this comment

Ericson2314 Jun 23, 2018 • edited

Choose a reason for hiding this comment

matthewbauer Jun 23, 2018

Choose a reason for hiding this comment

Ericson2314 Jun 23, 2018

Choose a reason for hiding this comment

matthewbauer commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 23, 2018

GrahamcOfBorg commented Jun 25, 2018

Ericson2314 Jun 25, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

GrahamcOfBorg commented Jun 25, 2018

layus commented Jun 26, 2018

matthewbauer commented Jun 23, 2018 •

edited by Ericson2314

Ericson2314 Jun 23, 2018 •

edited