Mobile updates #50596

svanderburg · 2018-11-18T14:45:46Z

Motivation for this change

Replaces the androidenv implementation by a new implementation that uses:

The latest repository specs to generate expressions from (repository2-1.xml)
A more robust generation strategy that exposes many more plugins
A more robust autopatching strategy for ELF binaries
A function composition strategy that makes it easier to maintain androidenv.buildApp and the corresponding Android SDK features a build may need

Furthermore, it updates other mobile build aspects:

A refactored xcodeenv implementation (iphone build environment)
A refactored titaniumenv implementation using the new androidenv and xcodeenv environments

Status:

Done:

New androidenv implementation
Refactored xcodeenv implementation
Refactored titaniumenv implementation
Added docs for androidenv to the manual
Added docs for xcodeenv to the manual
Added docs for titaniumenv to the manual

Broken packages ported:

Still to do:

Use modified autopatchelfHook that can also be used as a CLI
Add license accepted flag, because the Android SDK requires you to accept a license agreement
Readd androidenv.buildGradleApp
Fix cross compilation (stdenv.cross) and androidPkgs

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

nixos-discourse · 2018-11-18T14:47:09Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/my-revised-experimental-android-build-functions/923/12

svanderburg · 2018-11-18T14:47:41Z

@Ericson2314 I could use your help for the stdenv.cross

svanderburg · 2018-11-18T14:48:39Z

@matthewbauer I could use your help for readding androidenv.buildGradleApp

pkgs/development/mobile/androidenv/autopatchelf/autopatchelf

pkgs/development/mobile/androidenv/ndk-bundle/make_standalone_toolchain.py_18.patch

svanderburg · 2018-11-20T22:04:33Z

@Mic92 I simplified the patch

Ericson2314 · 2018-11-21T06:24:18Z

CC @luigy. We should probably have a nixpkgs-master branch of reflex platform so when we upgrade to the next stable release there are no surprises.

doc/languages-frameworks/ios.section.md

Ericson2314 · 2018-11-27T18:02:27Z

@svanderburg so I think the "android ndk packages" file I made you'll basically want to keep as-is. The point of it is to take the NDK stuff, however that is gotten, and chop it up into things that we can replace normal nixpkgs derivations with (gcc, bionic libc, etc). Maybe with your new finer-grained plugin approach, it will be easier to get just those bits rather than downloading more and then throwing array the rest, but most of the complicatedness in that file has to do with the fact that tools and library overrides belong in different bootstrapping stages, and that much remains the same.

svanderburg · 2018-11-27T20:18:47Z

I just rebased against the latest master. This resolves a conflict and allows me to use the modified autoPatchelf executable implemented in issue #50802

svanderburg · 2018-11-27T20:21:21Z

@Ericson2314 Thanks! I'll start investigating a reintegration approach of these package soon.

The stdenv/cross environment is also broken because of the new packaging structure -- we now need to refer to the Android NDK (that is now bundled as a plugin with the Android SDK) in a different way.

svanderburg · 2018-11-28T15:04:20Z

@Ericson2314 I did a preliminary investigation and there are some practical concerns for which I need to know your opinion.

The androidndk-pkgs.nix file is basically a set of packages that you want to cross compile with the Android NDK, right?

I'm not sure if putting this file in androidenv/ is the most logical place. For example, packages built with OpenJDK or Python are also not technically in the same "environment". The purpose of the androidenv component is to encapsulate the Android SDK manager and its plugins.

Do you mind if I put this expression in pkgs/development/androidndk-packages ? This is a location where also packages for other kinds of development environments reside, e.g. pkgs/development/python-modules/, pkgs/development/node-modules/ etc. I think this also makes it easier to implement future refactorings of the androidenv

The other problem is the Android NDK package -- the newest Android SDK also bundles the NDK as a plugin that you can install with the SDK manager. This is how I integrate the Android NDK in Nixpkgs.

I noticed that stdenv/cross supports various kinds of NDK versions. Unfortunately, the repository2-1.xml config file provided by Google (that I use in the new generation process) only provides one NDK version, namely: 18b

Is it still desired to support older NDK versions? If so, then we must package these older NDK versions differently somehow, by using the XML files of older SDK versions.

Ericson2314 · 2018-11-28T16:25:28Z

Firstly, it's fine to drop the other versions. I'd just ask that the code stay structured so that multiple SDK/NDK versions be supported in principle, so that once Google has more newstyle releases they can be simultaneously supported, but I assume you're doing that anyways :).

I don't mind if you move this file, but I don't think the comparison to the language packages is quite correct. The Android packages here are not, like, Java apps or something, but regular packages using prebuilt binaries. In other words the packages aren't really Android specific, only the platform they are built for and their provenance is. Language packages OTOH are different packages, but built in as normal a Nix approach as possible. That's almost the opposite.

I do appreciate trying to get it away from any autogenerated files, though, so move as you see fit.

svanderburg · 2018-11-28T19:50:07Z

@Ericson2314 I now started investigating whether I can separate the androidndkPackages from androidenv (on the upstream master branch).

The idea is that I first try to separate it and check whether the functionality still works (e.g. whether there are no regressions). If this succeeds, then I apply the patches that provide my new SDK implementation. Finally, I will fix the androidndk packages to work with the NDK bundled with the new SDK implementation (version 18b).

Is there simple way to test how this cross compiling stuff works? I noticed that there is bionic in all-packages.nix, but I can't seem to build it by running:

$ nix-build -A bionic

The above command throws an assertion error.

I probably need to supply a bunch of cross compiling parameters, but from the documentation it is not clear to me how I can, for example, build on my x86_64-linux system a bionic implementation for ARM or x86_64 (or whatever targets are supported).

Is there simple testcase that I can run to make bionic build properly?

Ericson2314 · 2018-11-28T19:56:46Z

$ nix-build -A pkgsCross.aarch64-android-prebuilt.hello

will build GNU hello with ndk's GCC and bionic.

[I pinged you on IRC if you have any follow-up questions.]

svanderburg · 2018-11-28T21:32:20Z

@Ericson2314 I just created a new branch (forked from the current Nixpkgs) in which I implemented the separation (svanderburg@13d6088). Your testcase seems to pass.

What I basically did: I moved androidPkgs from androidenv to the top-level expression and moved all androidPkgs stuff to pkgs/development/androidndk-pkgs.

Next thing is trying to rebase my SDK changes on top of this change and fixing to make it work with the Android NDK 18b.

Ericson2314 · 2018-11-28T22:30:17Z

Looks great! And btw, thank you for maintaining this.

svanderburg · 2018-12-01T15:35:25Z

I just rebased again to include the android NDK pkgs separation.

@Ericson2314 I have managed to partially fix the cross compilation, but I need some help from you. The example package that you provided me now seems to properly evaluate. However, when I run the build, but the configure script still fails to properly detect the compiler. Inspecting config.log shows me that it can't find -lgcc, which makes sense since the newest Android NDK is clang-based.

I have created a separate WIP branch (https://github.com/svanderburg/nixpkgs/tree/mobile-updates-wip) that contains my changes so far: svanderburg@63f235c

I think quite a few things in the Android 18b NDK have changed compared to the latest previous version 17c. For example, symlinking to the cross compilation executables no longer seems to work because they are shell script that themselves are wrappers using relative paths.

Moreover, since the newest Android SDK distribution only supports NDK 18b so far, and the older NDK versions are no longer supported, we can probably also remove the code that is specific to these older version, e.g. 17c, 10e, 8e.

Can you have look at it?

Ericson2314 · 2018-12-01T18:00:38Z

I wonder if -lgcc has something to do with one of our wrapper scripts. hello itself ought not to care about those details.

svanderburg · 2018-12-18T22:05:51Z

@Ericson2314 I think I have managed to make the fixes on my own. Apparently, what I learned is that libgcc.a is still required in a clang build if GNU Binutils are used for linking. I modified the bintools package in such a way that the required libgcc.a can be found.

Another problem I ran into is that the asm/types.h header can no longer be found when I try to compile GNU Hello with bionic. This is most likely caused by a slightly different sysroot directory organization in the Android NDK. I modified the bionic expression in such a way that these target specific ASM headers are copied in the right locations.

After making these modifications I have successfully compiled GNU Hello for aarch64 on Android.

Since I'm not using this stuff extensively, let me know if there is anything else that may require fixing. My changes are in revision 5142862

I think everything now is fixed and there are, as far as I can see, no breaking changes so I believe there is nothing preventing me from removing the WIP status of this pull request.

Ericson2314 · 2018-12-19T04:41:23Z

Great job! And sorry I didn't get to it before. I am taking a look now. BTW since you are doing xcode stuff, you may also want to checkout out what we did for ios cross here: https://github.com/nixos/nixpkgs/blob/master/pkgs/os-specific/darwin/xcode/default.nix using https://github.com/nixos/nixpkgs/blob/master/pkgs/os-specific/darwin/ios-sdk-pkgs/default.nix for the cross packages.

Ericson2314 · 2018-12-19T05:24:02Z

pkgs/development/androidndk-pkgs/androidndk-pkgs.nix

@@ -54,8 +56,15 @@ rec {
      mkdir -p $out/bin
      for prog in ${ndkBinDir}/${targetInfo.triple}-*; do
        prog_suffix=$(basename $prog | sed 's/${targetInfo.triple}-//')
-        ln -s $prog $out/bin/${stdenv.targetPlatform.config}-$prog_suffix
+        cat > $out/bin/${stdenv.targetPlatform.config}-$prog_suffix <<EOF


Any particular reason you switched from a here-doc?

Yes, symlinking does not work for the executables in Android NDK 18b. The executables we refer to are bash scripts that load executable from relative locations and have the expectation that they are loaded from the Android NDK directory. If we use symlinks, then it will look relative to the path of the symlinks, breaking the executable because these paths are invalid.

Ericson2314 · 2018-12-19T05:24:32Z

pkgs/development/androidndk-pkgs/androidndk-pkgs.nix

    in
    runCommand "bionic-prebuilt" {} ''
      mkdir -p $out
      cp -r ${includePath} $out/include
+      chmod u+w $out/include
+      cp -r ${asmIncludePath}/* $out/include
      chmod +w $out/include


This chmod now happens twice?

Hmm is not needed. Basically I needed to restore write permissions first before I could copy the asm headers, but I didn't bother to change the original statement that restores write permissions

svanderburg · 2018-12-19T21:48:28Z

@Ericson2314 I just removed the redundant chmod instruction. I also removed the WIP status from this pull request.

I will give people just a little while to provide their feedback. I think these changes can be integrated soon. We can always refine the infrastructure after we have integrated it if something appears to be missing.

svanderburg · 2018-12-23T14:36:33Z

Since I have not received any further comments, I'm planning to integrate this PR tomorrow

7c6f434c · 2018-12-26T14:12:59Z

Hm. It looks like adb per se has Apache2 source, and previously it was provided as a free package (while the SDK itself was unfree). I am trying to keep unfree stuff in a separately controlled user-owned package set and I want adb in a free and root-owned package set. Is it currently possible?

nixos-discourse · 2018-12-29T09:09:07Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/building-openwrt-with-nix-discuss-on-35c3/1777/2

kirelagin · 2019-01-20T18:25:30Z

I’m not sure if it is this PR that changed it, but it looks like adb and other useful tools end up in libexec instead of bin in platform-tools, so they are no longer available on PATH.

Is this a bug, or is there some other recommended way of installing adb?

bkchr · 2019-05-23T09:02:03Z

@svanderburg Did you ever tried the ndk-build? For me it is failing :( It seems to pick up the system ld.gold linker. Any ideas on how to fix this?

Artturin · 2023-09-08T19:25:41Z

buildGradleApp was never readded

nixos-discourse · 2023-09-08T19:28:14Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-androidenv-error-functionargs-requires-a-function/32773/5

Artturin · 2023-09-08T19:39:55Z

Readding in #254082

GrahamcOfBorg added the 8.has: documentation label Nov 18, 2018

Mic92 reviewed Nov 19, 2018

View reviewed changes

pkgs/development/mobile/androidenv/autopatchelf/autopatchelf Outdated Show resolved Hide resolved

Mic92 requested changes Nov 19, 2018

View reviewed changes

aszlig mentioned this pull request Nov 19, 2018

autoPatchelfHook: Fixes/improvements for Android SDK emulator #50802

Merged

malbarbo reviewed Nov 21, 2018

View reviewed changes

doc/languages-frameworks/ios.section.md Outdated Show resolved Hide resolved

svanderburg force-pushed the mobile-updates branch from dcdcd38 to ae557d5 Compare November 27, 2018 20:17

svanderburg force-pushed the mobile-updates branch from 1a37262 to 1572b13 Compare December 1, 2018 15:09

svanderburg requested review from Ericson2314 and matthewbauer as code owners December 1, 2018 15:09

GrahamcOfBorg added the 6.topic: stdenv Standard environment label Dec 1, 2018

GrahamcOfBorg added the 2.status: merge conflict label Dec 4, 2018

svanderburg force-pushed the mobile-updates branch from 5dd3e24 to 15299d8 Compare December 5, 2018 19:29

GrahamcOfBorg removed the 2.status: merge conflict label Dec 5, 2018

svanderburg requested a review from infinisil as a code owner December 5, 2018 20:55

svanderburg added 4 commits December 18, 2018 21:16

adb-sync: fix to make it work with platform-tools

5e690b4

Remove redundant getBin function

b99ac1c

Reintroduce licenseAccepted option

3bfd3e7

Fix adb program module

8122431

svanderburg force-pushed the mobile-updates branch from 44f2f4c to 8122431 Compare December 18, 2018 20:21

Initial attempt to restore Android NDK cross building

5142862

svanderburg requested a review from nbp as a code owner December 18, 2018 21:58

Ericson2314 reviewed Dec 19, 2018

View reviewed changes

Remove redundant chmod

7a5a370

svanderburg changed the title ~~[WIP] Mobile updates~~ Mobile updates Dec 19, 2018

Remove relatedPackages to fix ofborg evaluation

e37f045

GrahamcOfBorg added 8.has: clean-up 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Dec 20, 2018

svanderburg mentioned this pull request Dec 21, 2018

Missing arguments when building the basic emulator in README.md svanderburg/nix-androidenvtests#5

Closed

svanderburg merged commit a27aa24 into NixOS:master Dec 24, 2018

ali-abrar mentioned this pull request Jul 10, 2019

Update to GHC 8.6.5 and nixpkgs 19.03 reflex-frp/reflex-platform#506

Merged

20 tasks

Artturin mentioned this pull request Sep 8, 2023

init androidenv.buildGradleApp #41855

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mobile updates #50596

Mobile updates #50596

svanderburg commented Nov 18, 2018 •

edited

nixos-discourse commented Nov 18, 2018

svanderburg commented Nov 18, 2018

svanderburg commented Nov 18, 2018

svanderburg commented Nov 20, 2018

Ericson2314 commented Nov 21, 2018

Ericson2314 commented Nov 27, 2018

svanderburg commented Nov 27, 2018

svanderburg commented Nov 27, 2018

svanderburg commented Nov 28, 2018

Ericson2314 commented Nov 28, 2018

svanderburg commented Nov 28, 2018

Ericson2314 commented Nov 28, 2018 •

edited

svanderburg commented Nov 28, 2018

Ericson2314 commented Nov 28, 2018 •

edited

svanderburg commented Dec 1, 2018

Ericson2314 commented Dec 1, 2018

svanderburg commented Dec 18, 2018

Ericson2314 commented Dec 19, 2018

Ericson2314 Dec 19, 2018

svanderburg Dec 19, 2018

Ericson2314 Dec 19, 2018

svanderburg Dec 19, 2018

svanderburg commented Dec 19, 2018 •

edited

svanderburg commented Dec 23, 2018

7c6f434c commented Dec 26, 2018

nixos-discourse commented Dec 29, 2018

kirelagin commented Jan 20, 2019

bkchr commented May 23, 2019

Artturin commented Sep 8, 2023

nixos-discourse commented Sep 8, 2023

Artturin commented Sep 8, 2023

Mobile updates #50596

Mobile updates #50596

Conversation

svanderburg commented Nov 18, 2018 • edited

Motivation for this change

Status:

Things done

nixos-discourse commented Nov 18, 2018

svanderburg commented Nov 18, 2018

svanderburg commented Nov 18, 2018

svanderburg commented Nov 20, 2018

Ericson2314 commented Nov 21, 2018

Ericson2314 commented Nov 27, 2018

svanderburg commented Nov 27, 2018

svanderburg commented Nov 27, 2018

svanderburg commented Nov 28, 2018

Ericson2314 commented Nov 28, 2018

svanderburg commented Nov 28, 2018

Ericson2314 commented Nov 28, 2018 • edited

svanderburg commented Nov 28, 2018

Ericson2314 commented Nov 28, 2018 • edited

svanderburg commented Dec 1, 2018

Ericson2314 commented Dec 1, 2018

svanderburg commented Dec 18, 2018

Ericson2314 commented Dec 19, 2018

Ericson2314 Dec 19, 2018

Choose a reason for hiding this comment

svanderburg Dec 19, 2018

Choose a reason for hiding this comment

Ericson2314 Dec 19, 2018

Choose a reason for hiding this comment

svanderburg Dec 19, 2018

Choose a reason for hiding this comment

svanderburg commented Dec 19, 2018 • edited

svanderburg commented Dec 23, 2018

7c6f434c commented Dec 26, 2018

nixos-discourse commented Dec 29, 2018

kirelagin commented Jan 20, 2019

bkchr commented May 23, 2019

Artturin commented Sep 8, 2023

nixos-discourse commented Sep 8, 2023

Artturin commented Sep 8, 2023

svanderburg commented Nov 18, 2018 •

edited

Ericson2314 commented Nov 28, 2018 •

edited

Ericson2314 commented Nov 28, 2018 •

edited

svanderburg commented Dec 19, 2018 •

edited