Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework androidenv package list generation #89775

Merged
merged 1 commit into from Jan 10, 2021

Conversation

numinit
Copy link
Contributor

@numinit numinit commented Jun 7, 2020

Motivation for this change

Google's Android repos duplicate package names. See #86043, #95512, #95805, and possibly others. Additionally, androidenv is unusable for building apps if you just export the tools in the environment to a shell.nix due to an android-sdk-license file not existing. See #23910.

Rewrite the generator script in Ruby/Nokogiri (as opposed to xsltproc) as it is easier to munge Google's repo XML the way we need. Commit a "repo.json" to the repository that is overridable as a parameter of composeAndroidPackages. Split generate.sh into fetchrepo.sh and mkrepo.sh for fetching the repo and writing out repo.json from the fetched repo XMLs respectively.

Add extraLicenses as a parameter of composeAndroidPackages.

Add repoJson and repoXmls as parameters of composeAndroidPackages. If repoJson is provided, androidenv will read the repo definitions from the specified JSON file.

If repoXmls are provided, androidenv will use the new generator script to build repoJson as a derivation.

Example shell.nix is now provided in examples/.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Possibly closes: #78623 #96205 #95512 #86043 #72220 #23910

@numinit

This comment has been minimized.

@numinit numinit marked this pull request as draft June 7, 2020 22:40
@numinit numinit requested a review from svanderburg June 7, 2020 22:40
@numinit numinit changed the title WIP: Update androidenv to allow duplicate package names and custom package definitions WIP: Update androidenv Jun 8, 2020
@tgunnoe
Copy link
Contributor

tgunnoe commented Jul 21, 2020

FYI I found lots of duplicates in the generator/* files, however I don't understand where these are "generated" from.

It looks to be a problem from converting to nix from the xml files as you mentioned above?

@numinit
Copy link
Contributor Author

numinit commented Jul 23, 2020

They're all generated from Google's repository XML using the generate script. Apparently they contain lots of duplicates...

@numinit
Copy link
Contributor Author

numinit commented Aug 22, 2020

I think I'm just going to write a script that spits out an intermediate derivation as a JSON file, then import it with builtins.fromJSON. The repo XML needs some processing.

@numinit numinit changed the title WIP: Update androidenv Update androidenv to avoid package name duplication Aug 22, 2020
@numinit numinit marked this pull request as ready for review August 22, 2020 21:28
@numinit
Copy link
Contributor Author

numinit commented Aug 22, 2020

This should be ready to go. There is a fully functional example that creates a FHS env with Android licenses accepted in examples/.

@numinit numinit changed the title Update androidenv to avoid package name duplication Update androidenv to add license hashes and avoid package name duplication Aug 22, 2020
@numinit numinit mentioned this pull request Aug 22, 2020
10 tasks
@numinit numinit mentioned this pull request Aug 23, 2020
@numinit
Copy link
Contributor Author

numinit commented Dec 12, 2020

Example has been updated to show how nixpkgs pinning works (i.e. using 20.09 nixpkgs with this androidenv)

@Mic92
Copy link
Member

Mic92 commented Dec 13, 2020

I re-applied your changes on top of master and I rerun this shell.nix that I used for home-assistant

{
  pkgs ? import <nixpkgs> {},
  pkgs_i686 ? pkgs.pkgsi686Linux,
  config ? pkgs.config
}:

let
  android = {
    versions = {
      tools = "26.1.1";
      platformTools = "30.0.5";
      buildTools = "29.0.2";
      ndk = "21.3.6528147";
      cmake = "3.10.2";
    };

    platforms = ["30"];
    abis = ["armeabi-v7a" "arm64-v8a"];
    extras = ["extras;google;gcm"];
  };

  androidEnv = pkgs.androidenv.override {
    inherit config pkgs pkgs_i686;
    licenseAccepted = true;
  };

  androidComposition = androidEnv.composeAndroidPackages {
    toolsVersion = android.versions.tools;
    platformToolsVersion = android.versions.platformTools;
    buildToolsVersions = [android.versions.buildTools];
    platformVersions = android.platforms;
    abiVersions = android.abis;

    includeSources = false;
    includeSystemImages = false;
    includeEmulator = false;

    includeNDK = true;
    ndkVersion = android.versions.ndk;
    cmakeVersions = [android.versions.cmake];

    useGoogleAPIs = true;
    includeExtras = android.extras;

    # If you want to use a custom repo JSON:
    #repoJson = ./repo.json;

    # If you want to use custom repo XMLs:
    /*repoXmls = {
      packages = [ ../xml/repository2-1.xml ];
      images = [
        ../xml/android-sys-img2-1.xml
        ../xml/android-tv-sys-img2-1.xml
        ../xml/android-wear-sys-img2-1.xml
        ../xml/android-wear-cn-sys-img2-1.xml
        ../xml/google_apis-sys-img2-1.xml
        ../xml/google_apis_playstore-sys-img2-1.xml
      ];
      addons = [ ../xml/addon2-1.xml ];
    };*/

    # Accepting more licenses
    extraLicenses = [
      # Already accepted for you with accept_license = true.
      # "android-sdk-license"

      # These aren't, but are useful for more uncommon setups.
      "android-sdk-preview-license"
      "android-googletv-license"
      "android-sdk-arm-dbt-license"
      "google-gdk-license"
      "intel-android-extra-license"
      "intel-android-sysimage-license"
      "mips-android-sysimage-license"
    ];
  };

  androidSdk = androidComposition.androidsdk;
  platformTools = androidComposition.platform-tools;
  jdk = pkgs.jdk;
in
pkgs.mkShell rec {
  name = "androidenv-demo";
  buildInputs = [ androidSdk platformTools jdk pkgs.android-studio ];
  LANG = "C.UTF-8";
  LC_ALL = "C.UTF-8";
  JAVA_HOME = jdk.home;

  # Note: ANDROID_HOME is deprecated. Use ANDROID_SDK_ROOT.
  ANDROID_SDK_ROOT = "${androidSdk}/libexec/android-sdk";
  ANDROID_NDK_ROOT = "${ANDROID_SDK_ROOT}/ndk-bundle";

  # Ensures that we don't have to use a FHS env by using the nix store's aapt2.
  GRADLE_OPTS = "-Dorg.gradle.project.android.aapt2FromMavenOverride=${ANDROID_SDK_ROOT}/build-tools/${android.versions.buildTools}/aapt2";

  shellHook = ''
    # Add cmake to the path.
    export PATH="$(echo "$ANDROID_SDK_ROOT/cmake/${android.versions.cmake}".*/bin):$PATH"

    cat <<EOF > local.properties
# This file was automatically generated by nix-shell.
sdk.dir=$ANDROID_SDK_ROOT
ndk.dir=$ANDROID_NDK_ROOT
EOF
  '';
}

I did not have any problems with openjdk as the other guy - also on master we have now openjdk 14 anyway. However it seems that build-tools no longer seems to build:

$ nix-shell
...
builder for '/nix/store/f8n1vc139wv0v5c0jvfs2jklq9hgw6n5-build-tools-29.0.2.drv' failed with exit code 1; last 10 log lines:
  searching for dependencies of /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64/libaapt2_jni.so
  searching for dependencies of /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64/libc++.so
  searching for dependencies of /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64/libbcc.so
    libbcinfo.so -> found: /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64/libbcinfo.so
    libc++.so -> found: /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64/libc++.so
    libncurses.so.5 -> found: /nix/store/cpi29nfqvxy4xmqdkq61s72n6rywcbmg-ncurses-6.2-abi5-compat/lib/libncurses.so.5
    libtinfo.so.5 -> found: /nix/store/cpi29nfqvxy4xmqdkq61s72n6rywcbmg-ncurses-6.2-abi5-compat/lib/libtinfo.so.5
  setting RPATH to: /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64:/nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tool
s-29.0.2/libexec/android-sdk/build-tools/29.0.2/lib64:/nix/store/cpi29nfqvxy4xmqdkq61s72n6rywcbmg-ncurses-6.2-abi5-compat/lib:/nix/store/cpi29nfqvxy4xmqdkq61s72n6rywcbmg-ncurses-6.2-
abi5-compat/lib
  automatically fixing dependencies for ELF files
  searching for dependencies of /nix/store/fhnm15fg9qg059yxfb0wzpf301kh3d1l-build-tools-29.0.2/libexec/android-sdk/build-tools/29.0.2/i686-linux-android-ld
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/mv8xb494r6qax4js94wwmal09r6q7czi-androidsdk.drv' failed to build
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------- nix-build
1 dependencies of derivation '/nix/store/19hsxmby2x5nix5kn0a7fk8a8irmp9vx-androidenv-demo.drv' failed to build
$ nix log /nix/store/f8n1vc139wv0v5c0jvfs2jklq9hgw6n5-build-tools-29.0.2.drv > /tmp/build-tools.log

Here is the log: https://gist.github.com/Mic92/c45dc94da236e2b96264693b951ec03d
I have not looked closer into this, maybe set -x in the build script would throw some light on this. Right now there is no real error message why the unpacking fails.
It might be not related to your changes - however it seems to be a regression from the last time I run this. For reference this is the rebase I tested on: https://github.com/Mic92/nixpkgs/tree/failing-android

@numinit
Copy link
Contributor Author

numinit commented Dec 13, 2020

@Mic92 I was able to reproduce this. However, it fails on the SDK tools instead of build-tools for me, because the build-tools are binary cached for me locally. With a set -x on tools/26.nix, it seems to die in autoPatchelf for some reason. Was there recently a change to autoPatchelf?

@numinit
Copy link
Contributor Author

numinit commented Dec 13, 2020

Seems to be caused indirectly by #101142: #101142 (comment)

@numinit
Copy link
Contributor Author

numinit commented Dec 19, 2020

Looks like the new version of the NDK (22.0.7026061) requires libxml and liblldb.so.11git. I was able to resolve libxml with autoPatchelf. However, I now get this, even when adding lldb_11 to nativeBuildInputs:

autoPatchelfHook could not satisfy dependency liblldb.so.11git wanted by toolchains/llvm/prebuilt/linux-x86_64/bin/lldb
Add the missing dependencies to the build inputs or set autoPatchelfIgnoreMissingDeps=true

Any hints?

@numinit
Copy link
Contributor Author

numinit commented Dec 19, 2020

I'll resolve the conflict once I can update the NDK FWIW.

@numinit
Copy link
Contributor Author

numinit commented Dec 19, 2020

Looks like they ship LLDB in the newer NDKs so had to conditionally add a new search path to autoPatchelf.

  • Default emulator updated to 30.3.4
  • Default NDK updated to 22.0.7026061
  • Fix attribute not found issue when fetching system images that don't exist for certain combinations of API/image type/image ABI

@numinit
Copy link
Contributor Author

numinit commented Dec 19, 2020

FWIW I tested the build locally against 20.09 nixpkgs and things seem to work. Still waiting on #106830 to be merged for master. And for some reason it looks like a lot of the JDK packages aren't binary cached, and were taking ages to compile on my poor laptop.

@numinit numinit force-pushed the update-androidenv branch 2 times, most recently from fa92598 to d8b7de6 Compare December 25, 2020 21:03
@numinit
Copy link
Contributor Author

numinit commented Dec 25, 2020

Rebased against master with #106830, included shell.nix now works in-tree without any modifications. Merry Christmas!

@numinit numinit force-pushed the update-androidenv branch 2 times, most recently from e1fa99d to b9941e4 Compare December 25, 2020 21:08
@numinit
Copy link
Contributor Author

numinit commented Jan 4, 2021

Fixed merge conflicts.

androidenv did not previously write license files, which caused certain
gradle-based Android tools to fail. Restructure androidenv's list of
Android packages into a single repo.json file to prevent duplication
and enable us to extract the EULA texts, which we then hash with
builtins.hashString to produce the license files that Android gradle
tools look for.

Remove includeDocs and lldbVersions, as these have been removed
from the Android package repositories.

Improve documentation and examples.
@Mic92
Copy link
Member

Mic92 commented Jan 10, 2021

Sorry. Forgot about this PR. Will have a look now.

@Mic92 Mic92 merged commit 5189496 into NixOS:master Jan 10, 2021
@eyJhb
Copy link
Member

eyJhb commented Jan 10, 2021

Good work, nice to see this getting merged!

jakubgs added a commit to status-im/status-mobile that referenced this pull request Jan 10, 2021
This makes use of the refactoring of Android SDK done in:
NixOS/nixpkgs#89775

Which allows us to drop the use of our own fork of `nixpkgs`.

Android Upgrades:
* Build Tools - `29.0.2` to `30.0.3`
* Platform Tools - `29.0.6` to `30.0.5`
* NDK Bundle - `21.0.6113669` to `21.3.6528147`

Other Upgrades:
* Git - `2.28.0` to `2.29.2`
* Go - `1.14.7` to `1.14.13`
* Clojure - `1.10.1.645` to `1.10.1.763`
* NodeJS - `12.18.3` to `12.20.1`
* Yarn - `1.22.4` to `1.22.10`
* OpenJDK - `8u265-ga` to `8u272-b10`
* PatchElf - `0.11` to `0.12`
* CoreUtils - `8.31` to `8.32`

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit to status-im/status-mobile that referenced this pull request Jan 10, 2021
This makes use of the refactoring of Android SDK done in:
NixOS/nixpkgs#89775

Which allows us to drop the use of our own fork of `nixpkgs`.

Android Upgrades:
* Build Tools - `29.0.2` to `30.0.3`
* Platform Tools - `29.0.6` to `30.0.5`
* NDK Bundle - `21.0.6113669` to `21.3.6528147`

Other Upgrades:
* Git - `2.28.0` to `2.29.2`
* Go - `1.14.7` to `1.14.13`
* Clojure - `1.10.1.645` to `1.10.1.763`
* NodeJS - `12.18.3` to `12.20.1`
* Yarn - `1.22.4` to `1.22.10`
* OpenJDK - `8u265-ga` to `8u272-b10`
* PatchElf - `0.11` to `0.12`
* CoreUtils - `8.31` to `8.32`

Signed-off-by: Jakub Sokołowski <jakub@status.im>
@numinit
Copy link
Contributor Author

numinit commented Jan 11, 2021

Thanks @Mic92!

jakubgs added a commit to status-im/status-mobile that referenced this pull request Jan 11, 2021
This makes use of the refactoring of Android SDK done in:
NixOS/nixpkgs#89775

Which allows us to drop the use of our own fork of `nixpkgs`.

Android Upgrades:
* Build Tools - `29.0.2` to `30.0.3`
* Platform Tools - `29.0.6` to `30.0.5`
* NDK Bundle - `21.0.6113669` to `21.3.6528147`

Other Upgrades:
* Git - `2.28.0` to `2.29.2`
* Go - `1.14.7` to `1.14.13`
* Clojure - `1.10.1.645` to `1.10.1.763`
* NodeJS - `12.18.3` to `12.20.1`
* Yarn - `1.22.4` to `1.22.10`
* OpenJDK - `8u265-ga` to `8u272-b10`
* PatchElf - `0.11` to `0.12`
* CoreUtils - `8.31` to `8.32`

Signed-off-by: Jakub Sokołowski <jakub@status.im>
@numinit numinit deleted the update-androidenv branch January 12, 2021 02:03
jakubgs added a commit to status-im/status-mobile that referenced this pull request Jan 13, 2021
This makes use of the refactoring of Android SDK done in:
NixOS/nixpkgs#89775

Which allows us to drop the use of our own fork of `nixpkgs`.

Android Upgrades:
* Build Tools - `29.0.2` to `30.0.3`
* Platform Tools - `29.0.6` to `30.0.5`
* NDK Bundle - `21.0.6113669` to `21.3.6528147`

Other Upgrades:
* Git - `2.28.0` to `2.29.2`
* Go - `1.14.7` to `1.14.13`
* Clojure - `1.10.1.645` to `1.10.1.763`
* NodeJS - `12.18.3` to `12.20.1`
* Yarn - `1.22.4` to `1.22.10`
* OpenJDK - `8u265-ga` to `8u272-b10`
* PatchElf - `0.11` to `0.12`
* CoreUtils - `8.31` to `8.32`

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet