Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apps are too big (in size) #202

Closed
mid-kid opened this issue Feb 1, 2014 · 65 comments
Closed

Apps are too big (in size) #202

mid-kid opened this issue Feb 1, 2014 · 65 comments

Comments

@mid-kid
Copy link

mid-kid commented Feb 1, 2014

Every separate kivy app installed has an identical copy of the libraries. That makes every kivy app be about 20MB in size when only including the kivy library and it's dependencies. That's very big considering the actual content of the app.

There are several options to solve this problem I can think of:

  • Go the necessitas/sl4a way and have an app for the libraries only. Users will be annoyed.
  • Have every app include a piece of code to download precompiled libs (+ deps) to a .kivy folder in the sdcard. The sdcard needs to be available and have enough space.
  • Use tinypy (though I doubt the compatibility + the project is pretty much moribund)
  • Instead of extracting the python libraries from a zip, I think the python interpreter can load the zip directly. This will help a bit for size, but not for (startup) performance.
  • I think nuitka can include all the necessary scripts into a single binary, then you just need several .so files (libpython, sdl, etc) and in my crappy theory, that should work. I don't know if this will reduce the size much, but at least you get performance, so that's a plus, I guess.
  • Any ideas?

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

@brousch
Copy link
Contributor

brousch commented Feb 1, 2014

We rehash this problem every few months. As you've discovered, there is no
ideal solution.
On Feb 1, 2014 5:59 PM, "mid-kid" notifications@github.com wrote:

Every separate kivy app installed has an identical copy of the libraries.
That makes every kivy app be about 20MB in size when only including the
kivy library and it's dependencies. That's very big considering the actual
content of the app.

There are several options to solve this problem I can think of:

  • Go the necessitas way and have an app for the libraries only. Users
    will be annoyed.
  • Have every app include a piece of code to download precompiled libs
    (+ deps) to a .kivy folder in the sdcard. The sdcard needs to be available.
  • Use tinypy (though I doubt the compatibility + the project is pretty
    much morubund)
  • Instead of extracting the python libraries from a zip, I think the
    python interpreter can load the zip directly. This will help a bit for
    size, but not for (startup) performance.
  • I think nuitka can include all the necessary scripts into a single
    binary, then you just need several .so files (libpython, sdl, etc) and in
    my crappy theory, that should work. I don't know if this will reduce the
    size much, but at least you get performance, so that's a plus, I guess.
  • Any ideas?

Reply to this email directly or view it on GitHubhttps://github.com//issues/202
.

@akshayaurora
Copy link
Member

Every kivy app is not 20MB, look at the showcase. It's 6MB, you need to trim your app. Kivy itself comes under 5-6 mb as evident by the showcase.

@tito
Copy link
Member

tito commented Feb 1, 2014

  • Avoiding extraction of the application, and use the files directly from the apk. But native libraries expect a C path to be used with fopen() (ffmpeg / libpng / libjpeg / etc.). I made an experiment to redirect all of them to the android asset_ native API, it worked. No more unpack on the first start, and all the start are almost the same. If we store the pyo/data in the zip with compression, everytime you'll want to read it, it will be much slower. If we save them in the APK without compression, the starttime is identical as the current method, the APK is bigger, but it takes less space it total on the disk (APK + unpack in /sdcard (or /data in private mode)).
  • Compiling Python to C using nuikta or Cython generate bigger files = .pyo is smaller than a .so. If the goal is to reduce the size of the app, that's wrong :)
  • A strip tool. This would be for Python, not just for Kivy: C have "strip" that can remove all the unused functions / methods, because during the link of your application (ie, on iOS), it build a list of all the symbol used, and knows the one who are not used at all, and strip them. We don't have a such tool for Python. With a similar approach + exclude / inclusion symbol file, we could generate a directory with only the files / symbol used by the app.
  • A better usage of strip in C. Currently, only the debugging symbol are stripped. We might be able to strip some part of the Python interpreter that we don't use at all.
  • Reduce the bootstrap (python -v -m 'import kivy.uix.widget'), see all the modules imported, and check if reordering / avoiding to use some modules might help. This will end to reduce the size of the app, and be faster at boot.

And more, but all the things we currently explored have much more drawbacks than usability. The current state is a mix. It can always be improved, but must not be complex to use as well.

@tito
Copy link
Member

tito commented Feb 1, 2014

And yes, a minimal APK size is ~5MB. 20MB is already big. And most of the time, it's due to the data in your application, not Kivy.

@brousch
Copy link
Contributor

brousch commented Feb 1, 2014

It is 20MB after extraction. If you ;look in Settings->Apps you can dee
this size. Showcase is about 35MB.
On Feb 1, 2014 6:18 PM, "Mathieu Virbel" notifications@github.com wrote:

And yes, a minimal APK size is ~5MB. 20MB is already big. And most of the
time, it's due to the data in your application, not Kivy.

Reply to this email directly or view it on GitHubhttps://github.com//issues/202#issuecomment-33886483
.

@mid-kid
Copy link
Author

mid-kid commented Feb 2, 2014

I see I missed a point there:
I was pretty much talking about the size in the internal memory after extraction and moved to SD, as shown by the android system package manager in the downloaded apps tab, which may be a bit problematic on devices low on internal memory.
The app I got the numbers with is a self-compiled launcher with only kivy (master branch) + deps.

Please note that the package manager has calculated the size wrong since as far as I can remember. Just try to move an app to/from the SD and look at the total.

Sorry for the confusion.

@adamski
Copy link

adamski commented Jul 20, 2014

This is my main concern before delving into learning Kivy for mobile app development... I'm currently using JUCE and C++ but would rather be using Python for main part of the app.. however, app size and battery drain are concerns that I've seen raised about Kivy mobile apps.

@tshirtman
Copy link
Member

I do think 6-7Megs are acceptable sizes for an app these days, and a lot more if you have data, you rarely ever see less than one meg apps anymore, imho. For battery drain, i never witnessed anything about kivy that would cause that, i do have a few unity apps that makes my phone burn my hand and empty my battery pretty quick (nexus 4), nothing of that kind happened with any kivy app that i tested.

@mid-kid
Copy link
Author

mid-kid commented Jul 20, 2014

@tshirtman

It may not be much of a problem on some newer phones with more storage, but if you can make it a bit smaller, and in the process support more phones, why not?
It can clearly be optimized.
And yes, <1m apps are rare, but I'm not talking about that. There's a big difference between that, and 20M (which makes it to the top 10 biggest apps on my phone).

P.S. I'm not talking about performance.

@brousch
Copy link
Contributor

brousch commented Jul 20, 2014

"Why not?" If we could do it without a bunch of extra work we would. This
has not been solved, so it has not been done.
On Jul 20, 2014 7:34 AM, "mid-kid" notifications@github.com wrote:

@tshirtman https://github.com/tshirtman

It may not be much of a problem on some newer phones with more storage,
but if you can make it a bit smaller, and in the process support more
phones, why not?
It can clearly be optimized.
And yes, <1m apps are rare, but I'm not talking about that. There's a big
difference between that, and 20M (which makes it to the top 10 biggest apps
on my phone).

P.S. I'm not talking about performance.


Reply to this email directly or view it on GitHub
#202 (comment)
.

@mid-kid
Copy link
Author

mid-kid commented Jul 20, 2014

Please note the "if you can" ;)

@adamski
Copy link

adamski commented Jul 20, 2014

I'm seeing two sizes stated here - ±20mb and 6-7mb - is the smaller size the compressed installer one would download from the app store? For my needs (musical tuning app), comparable apps are around 500kb on the app store.. so in this case, Kivy is not for me, but I am considering it for building API consuming mobile apps at the company I work for, as in this case size should not be such an issue.

@brousch
Copy link
Contributor

brousch commented Jul 20, 2014

6MB is the smallest size for the APK. Once installed, it will be reported
as about 20MB in Android App Settings.
On Jul 20, 2014 10:22 AM, "Adam Wilson" notifications@github.com wrote:

I'm seeing two sizes stated here - ±20mb and 6-7mb - is the smaller size
the compressed installer one would download from the app store? For my
needs (musical tuning app), comparable apps are around 500kb on the app
store.. so in this case, Kivy is not for me, but I am considering it for
building API consuming mobile apps at the company I work for, as in this
case size should not be such an issue.


Reply to this email directly or view it on GitHub
#202 (comment)
.

@mid-kid
Copy link
Author

mid-kid commented Jul 20, 2014

A kivy APK is 6MB, once it is run for the first time, libraries are extracted, which ends up as 20MB (Compressed data + estracted data). I can confirm that the android app manager is not the only who reports so, diskusage does too.

@akshayaurora
Copy link
Member

@mid-kid. This is a known limitation. Kivy might not be suited for every
purpose out there. there have been some attempts at removing some more
default modules of python that are usually not needed like distuils etc.

Expecting the kivy apk to be around 500KB though might not be feasible, yet!

We need a good selection mechanism they while packaging figures out all the
libs needed and only includes those files that are actually being used.

On Sun, Jul 20, 2014 at 8:38 PM, mid-kid notifications@github.com wrote:

A kivy APK is 6MB, once it is run for the first time, libraries are
extracted, which ends up as 20MB (Compressed data + estracted data). I can
confirm that the android app manager is not the only who reports so,
diskusage does too.


Reply to this email directly or view it on GitHub
#202 (comment)
.

@knappador
Copy link
Contributor

Can we delete the zip after extraction?

On Mon, Jul 21, 2014 at 1:02 AM, Akshay Arora notifications@github.com
wrote:

@mid-kid. This is a known limitation. Kivy might not be suited for every
purpose out there. there have been some attempts at removing some more
default modules of python that are usually not needed like distuils etc.

Expecting the kivy apk to be around 500KB though might not be feasible,
yet!

We need a good selection mechanism they while packaging figures out all
the
libs needed and only includes those files that are actually being used.

On Sun, Jul 20, 2014 at 8:38 PM, mid-kid notifications@github.com
wrote:

A kivy APK is 6MB, once it is run for the first time, libraries are
extracted, which ends up as 20MB (Compressed data + estracted data). I
can
confirm that the android app manager is not the only who reports so,
diskusage does too.


Reply to this email directly or view it on GitHub
<
https://github.com/kivy/python-for-android/issues/202#issuecomment-49549502>

.


Reply to this email directly or view it on GitHub
#202 (comment)
.

@mid-kid
Copy link
Author

mid-kid commented Jul 20, 2014

What is it with people thinking I expect it to be under 1M? xD
Anyway, yes, @akshayaurora, that is why I opened this issue.
And I am too lazy to try myself, that's why I'm not expecting anything.

@knappador, no, the zip is contained inside the APK, and as such, it can't be removed.

We can try to remove some unneeded libs, but I don't think that would get very far. Some stripping could do something, like @tito said (I haven't looked at this project in some time now, so I don't know about the state of affairs), but I think the key resides in changing the distribution method of the libraries. No option proposed so far is ideal (imo neither is the current one).

Funny how much activity after 6 months.

@knappador
Copy link
Contributor

The possibility of not including everything in the .mp3 files is interesting and would be more effective than stripping ever could be. The entire tar/mp3 trick was done specifically to deal with the Android toolchain deleting files that we wanted to include in the APK. Maybe we can switch to only packaging files that the Android toolchain mistakenly deletes.

I'm currently looking into a data persistence issue when using --private. I bet the two problems cover the same ground.

@hamedmohammadi
Copy link

what aboat to use nuitka and compile all application into c and then use directly into ndk ? it has some optimazation like pack all used package(in my reading in the site)

@mid-kid
Copy link
Author

mid-kid commented Sep 7, 2014

If you read the thread, that question is already answered.
Yes it is possible, yes it will speed it up a bit, no it won't reduce the size, no it ain't easy (probably a lot of compile errors).
El dia 07/09/2014 10.36, hamed mohammadi notifications@github.com va escriure:what aboat to use nuitka and compile all application into c and then use directly into ndk ? it has some optimazation like pack all used package(in my reading in the site)

—Reply to this email directly or view it on GitHub.

@tito
Copy link
Member

tito commented Oct 1, 2014

Lot of activity here. One more addition to the thread:

  • Any form of C compilation cannot be used directly from a ZIP/APK, and it will be extracted somehow: if you put the C result library in the libs -> the phone will depack it at the installation. If you put the C result library within another path, you'll need to extract it. Because dlopen() internally need the filename. It cannot be something within an APK.
  • 500kb library with only Python and a start code, it might be possible, but with another implementation that CPython i guess, and kivy it out of the scope.
  • Yes, python can start to look within a zip. But remember, the APK itself is a zip. So if you use the files zipped in the apk, you need to read a zip of a zip. Or store the files correctly. But loading assets (images, audio, video) with low-level library need work, and tricks that i'm absolutely not confident too.

I'm currently experimenting a new way to boot Python application (not related to Kivy yet): bootstrap. The minimal bootstrap is based on NDK, extract only one file, setup the path to include zip import, have a mechanism to load CPython extensions library from the libs/ directory (so unpacked during the installation). Boot to python is really faster. BUT i wonder how far we can push this approach with all the python library.

the .egg format separate the libraries from the python code. This is interesting to use if we want to separate thoses in the final apk. However, the real issue is the data shipped with any python library. If you take Kivy, it will search for the glsl shaders in its own data/glsl directory. But it except a posix path, not a path within a zip. Same for images. Now try to think about a method that could work for any python packages: i didn't find one yet.

@hamedmohammadi
Copy link

tito your last answer was Great!!! may be it give some hope to use embedding python in widder situations. thanks a lot

@hamedmohammadi
Copy link

i dont know it can be helpfull but if there is a way to use jython in android we can add it as a jar file and we reduce the size of python itself

@mid-kid
Copy link
Author

mid-kid commented Jan 4, 2015

Jython does not support python3, and afaik, it does not work with native libraries compiled for CPython, such as SDL.

@endolith
Copy link

Kivy sounds perfect, being Python and cross-platform, but then all the apps it creates are huge for what they do, which makes me feel like it's not worth learning.

Every kivy app is not 20MB, look at the showcase. It's 6MB, you need to trim your app. Kivy itself comes under 5-6 mb as evident by the showcase.

On Android it's 37 MB:

kivy showcase 37 mb

How do you "trim your app"? Is this not automatic?

@inclement
Copy link
Member

@endolith: @akshayaurora was referring to the .apk file size (where 6-7MB is the minimum), not the unpacked size (where the ~30MB you see is a normal unpacked size for that minimal APK).

By trimming the app, I think @akshayaurora means removing parts of the python installation (and potentially our own app) that aren't necessary. Some such things are done automatically, where modules are unlikely to ever be useful on android, but doing it on a large scale would be extremely difficult as it's hard to inspect a python app and work out what modules it will need. Even doing this manually, I suspect it would be difficult to cut more than 1-2MB from the APK (though that's more difference when extracted).

@J-qak
Copy link

J-qak commented Jan 30, 2016

One utopic solution to this would be to have Python+Kivy commonly preinstaled on Android systems :-3

@v3ss0n
Copy link

v3ss0n commented Feb 1, 2016

make an android virus that automatically spreads and install Kivy , Cordova + crosswalk , etc :P

@mardon86
Copy link

Is there a way to create apk that contain only python script and additional module (without including python vm + kivy) ? and then have the apk detect if the python+kivy is installed. if python+kivy is not installed then download the python+kivy. Therefore every kivy app apk size will not too big.

@inclement
Copy link
Member

inclement commented Mar 28, 2017 via email

@KeyWeeUsr
Copy link
Contributor

@cdb1019 I have a small list of files&folders that could be removed from the Python packaged into APK archive, but a testing is needed:

folders:

  • distutils
  • email (if not needed)
  • xml (if not needed)

files:

  • pdb.doc
  • pdb files
  • profiling files
  • this
  • timeit
  • SMTP

maybe:

  • pprint (never used it directly)

That kills another ~1.86MB from unpacked Python (~7.5MB for me). The rest is only about cutting off stuff from Kivy that you don't need such as behaviors, complex widgets, kivy logos, keyboard layouts, fonts, effects, extras, recorder(input), some modules, storage (if you don't use it), tools (aren't packaged, I think). For all that you'll need to check if they aren't imported or referenced somewhere else (and that's a matter of change always). We can't really strip the Kivy parts as we don't have a clue what a user might use, but we might be able to strip Python a little bit more in the future.

@kamathln
Copy link

kamathln commented Apr 10, 2017

Anyone has plans to make an Android fork/mod/ROM with Kivy built in? - In such a way that all App developers need to do is package the main.py and main.kivy (obfuscated if necessary), and necessary support files not included with kivy. The install file also comes with a requirements.txt which the internal pip interprets.

Just a thought that crossed my mind. I don't see it growing big without going the way of Firefox OS. It is just for fun.

@mcroni
Copy link

mcroni commented Apr 10, 2017 via email

@kamathln
Copy link

kamathln commented Apr 11, 2017

Yes, It is purposefully vague at the moment. It is more of a problem statement than the solution. The final aim is to be able to distribute APKs or may be some other installer without the whole load of the Interpreter + libraries in every APK/installer. Also, the required python libraries can be installed centrally to reduce duplication.

But on giving it proper thought, I realized the duplication problem can't be correctly solved , as upgrading a library for one app would probably break other apps.

@kamathln
Copy link

Now this comment has nothing to do with the above idea. Have we tried experimenting with xz/lzma compression instead of zip? I know, when the app is decompressed initially it is finally going to occupy the same space, but at least it can reduce the compressed size?

@frmdstryr
Copy link

Perhaps a modified version of cx_Freeze could be used?

With cx_Freeze, you tell it which modules are needed for your app (already in the kivy requirements), then it scans all those modules and finds their dependencies (you can add ones it can't find automatically) and packs ONLY the needed python modules and their dependencies into a single library.zip file. Also it renames all .so files (only from required modules / packages) in the format package.name.so and places them in a single folder.

build/library.zip # all data, py/pyc files
build/libx.so
build/libpython.so
build/my.pkg.so
etc...

Once this is done:

  1. The library.zip could be extracted and packaged in the apk (unzipped). Since the apk is a zip, if the apk were added to the python path, the python files could then be imported directly with the standard zipimporter (maybe would need a modified path).
  2. All the necessary .so files could be copied to the lib folder so they are extracted upon install.

Pros:

  1. Automatically remove all unused modules (thus reducing overall footprint, i've seen 3x-4x reductions on desktop apps)
  2. Save space of unpacked python assets (current cost is tar.gz size + unpacked size) as they would be loaded right from the apk (excluding .so files)
  3. Remove startup time cost of unpacking assets on first run (since this would be done at install)

Cons:

  1. Would need tested and implemented
  2. Adds a startup loading time due to importing from zip (Importing only the stdlib.zip from crystax only takes 89ms to load)
  3. Loader would need to know to import so files based on dotted name instead of path.

Edit:
I tested this by packing everything into a zip, putting the so files in a single location, hacking the import to load the correct so files, and the startup time was rather slow (extra 4 seconds) due to importing from the zipfile. So some alternative way needs investigated.

Perhaps extraction could be sped up using czipfile, or the library contents could be loaded into memfs or something.

@frmdstryr
Copy link

frmdstryr commented Jul 12, 2017

I found out how to have all the .so files copied during app install (not on first run!) and get loaded properly from the app.

Doing it this way you can support multiple arch's in one apk, EVEN when your python requirements has compiled components as only the actual py/pyc/pyo files are in the assets folder. You can also then compress all the python libs into a zip and import them from the zip (although it's slower at the moment) or extract as normal.

  1. You have to move all .so files to your libs/<arch>/ folder (where libs is the jniDir)
  2. Rename them to lib.<pkg>.<of>.module.so (if they dont start with lib, android won't copy them from the apk automatically)
  3. Add the module importer to your app before importing anything using the modules and add it with sys.meta_path.append(AndroidFinder())
  4. Add the APK_LIB_DIR to your environment, can be set via the jni and retrieved using getApplicationInfo().nativeLibraryDir in your activity.
'''
Copyright (c) 2017, Jairus Martin.

Distributed under the terms of the MIT License.

The full license is in the file COPYING.txt, distributed with this software.

Created on July 10, 2017

@author: jrm
'''
import os
import sys
import imp
from glob import glob


class AndroidLoader(object):
    """ Loads renamed .so files from the app's lib folder"""

    def load_module(self, mod):
        try:
            return sys.modules[mod]
        except KeyError:
            pass

        lib = AndroidFinder.so_modules[mod]
        m = imp.load_dynamic(mod, lib)
        sys.modules[mod] = m
        return m


class AndroidFinder(object):
    """ Loads renamed so files from the app's lib folder"""
    so_modules = {}

    def __init__(self):
        #: Find all included so files
        lib_dir = os.environ['APK_LIB_DIR']

        for lib in glob('%s/lib.*.so'%lib_dir):
            name = lib.split("/")[-1] # Lib filename
            mod = ".".join(name.split(".")[1:-1])  # Strip lib and so
            AndroidFinder.so_modules[mod] = lib


    def find_module(self, mod, path = None):
        if mod in self.so_modules:
            return AndroidLoader()
        return None

@kamathln
Copy link

@endolith I know I am more than 18months late .. But have you checked if the apk generated previously is also getting included in the app? When my app size started doubling every build , I noticed that the previous generated apk was getting bundled into the current apk ..

So, over the last few iterations, my development pattern has morphed somewhat .. I keep all the files that go into the real app in the src directory, and give p4a the src directory as the private folder. The blacklist.txt , project config and other files remain in the project's root directory. I suggest Kivy developers using python-for-android to switch to this pattern. Feels clean.

@kamathln
Copy link

@frmdstryr Please look into https://developer.android.com/studio/build/configure-apk-splits.html if you are interested in building multiple architectures. I myself haven't tried it, but is definitely on my todo list.

@inclement
Copy link
Member

@frmdstryr That's extremely encouraging, thanks for the research! I'll certainly try this

@frmdstryr
Copy link

frmdstryr commented Jul 22, 2017

I was able to get about a 15% reduction by using pyminifier.

Interestingly, it reduces the py files about about 50%, but when compiled to pyo the difference is only ~15%. Still, took the extracted python from 9.6 MB to 8.2 MB and reduced the compressed python by about 0.5MB (3.2MB to 2.7MB). So about 2MB overall, better than nothing.

'''
Copyright (c) 2017, Jairus Martin.

Distributed under the terms of the MIT License.

The full license is in the file COPYING.txt, distributed with this software.

Created on July 21, 2017

@author: jrm
'''

def minifiy_sources(ctx, src):
    """ Finds all .py files within a path and runs minification on them.
        this should be done BEFORE compileall is run.

        Warning: This does IN PLACE minification.
    """

    info("Minifying sources, this may take some time...")
    from pyminifier import token_utils, minification

    class Options(object):
        tabs = False  #: Replace indentation with tabs (vs single space)

    cwd = os.getcwd()
    original = 0
    minified = 0
    passed = 0
    total = 0
    with current_directory(src):
        #: Find all files
        for py in shprint(sh.find, '.', '-name', '*.py').split("\n"):
            py_file = realpath(join(src, py))
            if not exists(py_file) or isdir(py_file):
                continue
            total +=1
            #: Read source code
            with open(py_file) as f:
                code = f.read()
                original += len(code)
            try:
                #: Minify code
                #: The operators seem to have a bug...
                minified_code = code
                for f in [minification.remove_comments_and_docstrings,
                          minification.fix_empty_methods,
                          minification.remove_blank_lines,
                          minification.reduce_operators,
                          minification.dedent
                          ]:
                    minified_code = f(minified_code)

                #tokens = token_utils.listified_tokenizer(code)
                #minified_code = minification.minify(tokens, Options)
                #print("OK",)
            except Exception as e:
                info("Minifying {}... Failed! Reason: {}".format(py, e))
                #raise
                continue  # Can't minify, leave it as is!

            #: Write minified code
            with open(py_file, 'wb') as f:
                f.write(minified_code)

            #: Make sure it compiles
            try:
                #: Minification compiles to pyo to make sure it's ok
                #: Compile pip packages to bytecode
                sh.bash('-c',
                        "source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
                        "PYTHONPATH={} python -{} -m py_compile {}"
                        .format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))

                minified += len(minified_code)
                passed +=1
            except Exception as e:
                info("Compile {}... Failed! Reason: {}".format(py, e))
                #: Undo
                with open(py_file, 'wb') as f:
                    f.write(code)
            #     #raise
                minified += len(code)

                #: Optimize original file
                sh.bash('-c',
                        "source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
                        "PYTHONPATH={} python -{} -m py_compile {}"
                        .format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))

    info_main("Minification complete total={} passed={}({}%) savings={}%!"
         .format(total, passed, round(100*passed/total), round(100*minified/original)))

@KeyWeeUsr
Copy link
Contributor

@frmdstryr If you can, try only the minification on the whole Python's Lib folder recursively. Maybe the best would be not to include it to the recipe, but use only some console arg and do it before actually packaging the application i.e. right before the private.mp3 creation.

I remember trying pyminifier with pyinstaller and there it worked just fine even with the obfuscation, however on android we might get into troubles with that. I tried a similar thing with replacing 4 spaces with 1 tab, but the size change wasn't that significant for such a harsh operation and with Kivy itself the change was almost invisible + when you start mixing tabs & spaces in the app, well... :D In any case, minification might even backfire, so if included, we probably shouldn't do it automatically.

@kamathln
Copy link

kamathln commented Jul 22, 2017

@frmdstryr Has anyone researched on Ultimate Executable Packer ? The code is currently way out of my league .. can it be adapted for python pyc/pyo , etc.. ?

Might affect startup performance though.

@KeyWeeUsr
Copy link
Contributor

Every such compression will need decompression on the device. It doesn't really matter how quickly you compress if you end up with the loading screen displayed for a minute. You can't just make the whole interpreter disappear with compressing, you need to remove files from it to actually make a usable difference. Just saying.

@frmdstryr
Copy link

@KeyWeeUsr That's what I'm doing, except I'm using crystax python 2. Currently I extract the stdlib.zip and site-packages to one folder and run minify on all py files within it recursively so it minifies(?) everything except my app code files. I added a flag to p4a to enable/disable it.

In any case, minification might even backfire, so if included, we probably shouldn't do it automatically.

Yes, exactly. If it does get added, it should be disabled by default as it's more of a final product optimization anyways.

Minify does screw up some files, for instance the sre*.py modules used by re in the python Lib fails to import (even though it compiles to pyo fine?). I also have it skip the join_multiline_pairs minification as that seemed to cause a lot of issues. But some method of exclusion would also be needed because every user has different requirements.

@kamathln p4a already compresses everything with tar.gz so there won't be much of a gain there.

@frmdstryr
Copy link

It would be interesting if some sort of bundler could be used like what is commonly done with javascript (webpack for example). react-native does this on release builds. It wouldn't save space (unless you can do minification on the bundle) but you could use the trick to have it extracted during install and avoid the startup extraction.

Pros:

  1. Importing would be one file, which "could" potentially save a lot of system calls and speedup importing (theoretically, I have no data to back it up). Currently python itself is ~700 files which is a lot of blocking system calls.
  2. It could be extracted on install by tricking android into thinking it's a native lib, saving extraction time on first startup.

Cons:

  1. Would still need to extract any data files.
  2. Would need copied for each arch (if multiple are used)

The only "packer" I found was pinliner. I ran a few tests, and it can't bundle multiple packages and modules at the moment (but does work nice for a single package), and it doesn't pack pyc/pyo files. If these were implemented, the python Lib and site-packages could be packed into one big file, then it could be named something that tricks android into copying it with the native modules (ex lib.pymodules.so) during install and imported from there.

@kamathln
Copy link

kamathln commented Jul 26, 2017

@frmdstryr Thanks for correcting my assumption. Had assumed it is plain tar.

Tried re-compressing the tar.gz with tar.xz instead. My private.mp3 came down from 6.3mb to 4.7mb . The decompression speed was 3x slower than gz though. (still much faster than bz2) . The comparizon was on a laptop though. Planning to try on my phone using termux .

@kamathln
Copy link

kamathln commented Jul 26, 2017

On my phone, in Termux:

$ time bzcat t.tar.bz2 > /dev/null

real    0m3.672s
user    0m3.520s
sys     0m0.030s
$ time zcat t.tgz > /dev/null

real    0m0.489s
user    0m0.420s
sys     0m0.020s

$ time xzcat t.tar.xz > /dev/null

real    0m1.191s
user    0m1.110s
sys     0m0.040s
$

@kamathln
Copy link

kamathln commented Jul 26, 2017

The tar.gz is simply a rename of private.mp3 . tar.xz and tar.bz2 are created with ....

zcat t.tar.gz2 | xz > t.tar.xz

....technique

@tasen-sp
Copy link

Can I do it by just deleting blank lines, comments,unused things from kivy???

@inclement
Copy link
Member

No, you won't get much if any gain from that. We already compile and ship bytecode and that doesn't help the size much.

@tasen-sp
Copy link

tasen-sp commented Jul 21, 2020 via email

@tshirtman
Copy link
Member

the issue is not the size of python code, it's mostly the size of binaries, like python itself, and all the cpython extensions (like kivy, but also others), that can quickly run into the megabyte range.

I'm not sure this issue is actionable, it's too broad, there will always be reasons to argue that apps are "too big", i'd rather see someone analyse and point at specific issues that can be fixed to make the apk / installed size smaller, and to have these fixed independently. So i'm going to close it, if another maintainer disagree, feel free to reopen, if an user sees a specific issue to fix to help address the point of this issue, feel free to open another one describing the exact issue you see.

@omeco4christ
Copy link

Can this be achieved by removing unused libraries and packages?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests