New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework androidenv package list generation #89775
Conversation
This comment has been minimized.
This comment has been minimized.
pkgs/development/mobile/androidenv/compose-android-packages.nix
Outdated
Show resolved
Hide resolved
2cf9d56
to
3b7642d
Compare
pkgs/development/mobile/androidenv/generated/system-images-android.nix
Outdated
Show resolved
Hide resolved
pkgs/development/mobile/androidenv/generated/system-images-android.nix
Outdated
Show resolved
Hide resolved
pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
Outdated
Show resolved
Hide resolved
pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
Outdated
Show resolved
Hide resolved
pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
Outdated
Show resolved
Hide resolved
FYI I found lots of duplicates in the generator/* files, however I don't understand where these are "generated" from. It looks to be a problem from converting to nix from the xml files as you mentioned above? |
They're all generated from Google's repository XML using the generate script. Apparently they contain lots of duplicates... |
I think I'm just going to write a script that spits out an intermediate derivation as a JSON file, then import it with builtins.fromJSON. The repo XML needs some processing. |
3b7642d
to
d2e87c3
Compare
This should be ready to go. There is a fully functional example that creates a FHS env with Android licenses accepted in examples/. |
pkgs/development/mobile/androidenv/compose-android-packages.nix
Outdated
Show resolved
Hide resolved
Example has been updated to show how nixpkgs pinning works (i.e. using 20.09 nixpkgs with this androidenv) |
fb1c955
to
9567b53
Compare
I re-applied your changes on top of master and I rerun this shell.nix that I used for home-assistant {
pkgs ? import <nixpkgs> {},
pkgs_i686 ? pkgs.pkgsi686Linux,
config ? pkgs.config
}:
let
android = {
versions = {
tools = "26.1.1";
platformTools = "30.0.5";
buildTools = "29.0.2";
ndk = "21.3.6528147";
cmake = "3.10.2";
};
platforms = ["30"];
abis = ["armeabi-v7a" "arm64-v8a"];
extras = ["extras;google;gcm"];
};
androidEnv = pkgs.androidenv.override {
inherit config pkgs pkgs_i686;
licenseAccepted = true;
};
androidComposition = androidEnv.composeAndroidPackages {
toolsVersion = android.versions.tools;
platformToolsVersion = android.versions.platformTools;
buildToolsVersions = [android.versions.buildTools];
platformVersions = android.platforms;
abiVersions = android.abis;
includeSources = false;
includeSystemImages = false;
includeEmulator = false;
includeNDK = true;
ndkVersion = android.versions.ndk;
cmakeVersions = [android.versions.cmake];
useGoogleAPIs = true;
includeExtras = android.extras;
# If you want to use a custom repo JSON:
#repoJson = ./repo.json;
# If you want to use custom repo XMLs:
/*repoXmls = {
packages = [ ../xml/repository2-1.xml ];
images = [
../xml/android-sys-img2-1.xml
../xml/android-tv-sys-img2-1.xml
../xml/android-wear-sys-img2-1.xml
../xml/android-wear-cn-sys-img2-1.xml
../xml/google_apis-sys-img2-1.xml
../xml/google_apis_playstore-sys-img2-1.xml
];
addons = [ ../xml/addon2-1.xml ];
};*/
# Accepting more licenses
extraLicenses = [
# Already accepted for you with accept_license = true.
# "android-sdk-license"
# These aren't, but are useful for more uncommon setups.
"android-sdk-preview-license"
"android-googletv-license"
"android-sdk-arm-dbt-license"
"google-gdk-license"
"intel-android-extra-license"
"intel-android-sysimage-license"
"mips-android-sysimage-license"
];
};
androidSdk = androidComposition.androidsdk;
platformTools = androidComposition.platform-tools;
jdk = pkgs.jdk;
in
pkgs.mkShell rec {
name = "androidenv-demo";
buildInputs = [ androidSdk platformTools jdk pkgs.android-studio ];
LANG = "C.UTF-8";
LC_ALL = "C.UTF-8";
JAVA_HOME = jdk.home;
# Note: ANDROID_HOME is deprecated. Use ANDROID_SDK_ROOT.
ANDROID_SDK_ROOT = "${androidSdk}/libexec/android-sdk";
ANDROID_NDK_ROOT = "${ANDROID_SDK_ROOT}/ndk-bundle";
# Ensures that we don't have to use a FHS env by using the nix store's aapt2.
GRADLE_OPTS = "-Dorg.gradle.project.android.aapt2FromMavenOverride=${ANDROID_SDK_ROOT}/build-tools/${android.versions.buildTools}/aapt2";
shellHook = ''
# Add cmake to the path.
export PATH="$(echo "$ANDROID_SDK_ROOT/cmake/${android.versions.cmake}".*/bin):$PATH"
cat <<EOF > local.properties
# This file was automatically generated by nix-shell.
sdk.dir=$ANDROID_SDK_ROOT
ndk.dir=$ANDROID_NDK_ROOT
EOF
'';
} I did not have any problems with openjdk as the other guy - also on master we have now openjdk 14 anyway. However it seems that build-tools no longer seems to build:
Here is the log: https://gist.github.com/Mic92/c45dc94da236e2b96264693b951ec03d |
@Mic92 I was able to reproduce this. However, it fails on the SDK tools instead of build-tools for me, because the build-tools are binary cached for me locally. With a set -x on tools/26.nix, it seems to die in autoPatchelf for some reason. Was there recently a change to autoPatchelf? |
Seems to be caused indirectly by #101142: #101142 (comment) |
Looks like the new version of the NDK (22.0.7026061) requires libxml and
Any hints? |
I'll resolve the conflict once I can update the NDK FWIW. |
9567b53
to
b84823b
Compare
Looks like they ship LLDB in the newer NDKs so had to conditionally add a new search path to autoPatchelf.
|
FWIW I tested the build locally against 20.09 nixpkgs and things seem to work. Still waiting on #106830 to be merged for master. And for some reason it looks like a lot of the JDK packages aren't binary cached, and were taking ages to compile on my poor laptop. |
fa92598
to
d8b7de6
Compare
Rebased against master with #106830, included shell.nix now works in-tree without any modifications. Merry Christmas! |
e1fa99d
to
b9941e4
Compare
b9941e4
to
9c8fc1d
Compare
Fixed merge conflicts. |
androidenv did not previously write license files, which caused certain gradle-based Android tools to fail. Restructure androidenv's list of Android packages into a single repo.json file to prevent duplication and enable us to extract the EULA texts, which we then hash with builtins.hashString to produce the license files that Android gradle tools look for. Remove includeDocs and lldbVersions, as these have been removed from the Android package repositories. Improve documentation and examples.
9c8fc1d
to
5b91d4a
Compare
Sorry. Forgot about this PR. Will have a look now. |
Good work, nice to see this getting merged! |
This makes use of the refactoring of Android SDK done in: NixOS/nixpkgs#89775 Which allows us to drop the use of our own fork of `nixpkgs`. Android Upgrades: * Build Tools - `29.0.2` to `30.0.3` * Platform Tools - `29.0.6` to `30.0.5` * NDK Bundle - `21.0.6113669` to `21.3.6528147` Other Upgrades: * Git - `2.28.0` to `2.29.2` * Go - `1.14.7` to `1.14.13` * Clojure - `1.10.1.645` to `1.10.1.763` * NodeJS - `12.18.3` to `12.20.1` * Yarn - `1.22.4` to `1.22.10` * OpenJDK - `8u265-ga` to `8u272-b10` * PatchElf - `0.11` to `0.12` * CoreUtils - `8.31` to `8.32` Signed-off-by: Jakub Sokołowski <jakub@status.im>
This makes use of the refactoring of Android SDK done in: NixOS/nixpkgs#89775 Which allows us to drop the use of our own fork of `nixpkgs`. Android Upgrades: * Build Tools - `29.0.2` to `30.0.3` * Platform Tools - `29.0.6` to `30.0.5` * NDK Bundle - `21.0.6113669` to `21.3.6528147` Other Upgrades: * Git - `2.28.0` to `2.29.2` * Go - `1.14.7` to `1.14.13` * Clojure - `1.10.1.645` to `1.10.1.763` * NodeJS - `12.18.3` to `12.20.1` * Yarn - `1.22.4` to `1.22.10` * OpenJDK - `8u265-ga` to `8u272-b10` * PatchElf - `0.11` to `0.12` * CoreUtils - `8.31` to `8.32` Signed-off-by: Jakub Sokołowski <jakub@status.im>
Thanks @Mic92! |
This makes use of the refactoring of Android SDK done in: NixOS/nixpkgs#89775 Which allows us to drop the use of our own fork of `nixpkgs`. Android Upgrades: * Build Tools - `29.0.2` to `30.0.3` * Platform Tools - `29.0.6` to `30.0.5` * NDK Bundle - `21.0.6113669` to `21.3.6528147` Other Upgrades: * Git - `2.28.0` to `2.29.2` * Go - `1.14.7` to `1.14.13` * Clojure - `1.10.1.645` to `1.10.1.763` * NodeJS - `12.18.3` to `12.20.1` * Yarn - `1.22.4` to `1.22.10` * OpenJDK - `8u265-ga` to `8u272-b10` * PatchElf - `0.11` to `0.12` * CoreUtils - `8.31` to `8.32` Signed-off-by: Jakub Sokołowski <jakub@status.im>
This makes use of the refactoring of Android SDK done in: NixOS/nixpkgs#89775 Which allows us to drop the use of our own fork of `nixpkgs`. Android Upgrades: * Build Tools - `29.0.2` to `30.0.3` * Platform Tools - `29.0.6` to `30.0.5` * NDK Bundle - `21.0.6113669` to `21.3.6528147` Other Upgrades: * Git - `2.28.0` to `2.29.2` * Go - `1.14.7` to `1.14.13` * Clojure - `1.10.1.645` to `1.10.1.763` * NodeJS - `12.18.3` to `12.20.1` * Yarn - `1.22.4` to `1.22.10` * OpenJDK - `8u265-ga` to `8u272-b10` * PatchElf - `0.11` to `0.12` * CoreUtils - `8.31` to `8.32` Signed-off-by: Jakub Sokołowski <jakub@status.im>
Motivation for this change
Google's Android repos duplicate package names. See #86043, #95512, #95805, and possibly others. Additionally, androidenv is unusable for building apps if you just export the tools in the environment to a shell.nix due to an
android-sdk-license
file not existing. See #23910.Rewrite the generator script in Ruby/Nokogiri (as opposed to xsltproc) as it is easier to munge Google's repo XML the way we need. Commit a "repo.json" to the repository that is overridable as a parameter of composeAndroidPackages. Split generate.sh into fetchrepo.sh and mkrepo.sh for fetching the repo and writing out repo.json from the fetched repo XMLs respectively.
Add extraLicenses as a parameter of composeAndroidPackages.
Add repoJson and repoXmls as parameters of composeAndroidPackages. If repoJson is provided, androidenv will read the repo definitions from the specified JSON file.
If repoXmls are provided, androidenv will use the new generator script to build repoJson as a derivation.
Example shell.nix is now provided in examples/.
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)Possibly closes: #78623 #96205 #95512 #86043 #72220 #23910