Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dist has gotten too large due to new maven features #2487

Closed
headius opened this issue Jan 20, 2015 · 18 comments
Closed

Dist has gotten too large due to new maven features #2487

headius opened this issue Jan 20, 2015 · 18 comments

Comments

@headius
Copy link
Member

headius commented Jan 20, 2015

The newish jar_dependencies feature in JRuby resulted in us having to ship Maven itself plus its dependencies. That increased the size of our distribution by more than 20MB. Other libraries contribute, but the ruby-maven-libs gem is the biggest issue.

Our users like the jar-dependencies feature, so it seems like we need to ship this stuff...but we need to shrink it down. Suggestions so far:

  • Install them as default gems. This will eliminate the cached gem file from our dist.
  • Use pack200 on the jars in the maven gem. That reduces size of the cached gem file (so moot point if we install as a default gem)
  • Install lazily or request user to install X.
@mkristian
Copy link
Member

did look at the complete distribution and used pack200 to reduce the jars which leads to:

$ ll -h *zip
41M  jruby-dist-9.0.0.0-SNAPSHOT-bin.zip
13M  jruby-dist-9.0.0.0-SNAPSHOT-src.zip
18M  jruby-dist-9.0.0.0-SNAPSHOT-pack200.zip

to use the pack200.zip you need to run a "post_install" script

find -name "*pack200.gz" | sed "s/\(.*\).pack200.gz/\1.pack200.gz \1.jar/" | xargs -L1 unpack200 -r

the reduction in size is impressive and worth for using with rbenv, rvm and alike. maybe even windows installer. the post_installer could be also an executable jar.

@mkristian
Copy link
Member

another point are the native libraries

  • both lib/jruby.jar has bundled the jni/* directory inside the jar and the dist archive contains lib/jni/* as well

@enebo
Copy link
Member

enebo commented Jan 23, 2015

I believe windows installer does use pack200 internally as part of install4j.

@mkristian
Copy link
Member

did some pack200 stuff on https://github.com/jruby/jruby/tree/pack20

jruby-dist-bin.tar.gz 37MB
jruby-dist-bin.zip 38MB
jruby-dist-bin200.tar.gz 17MB
jruby-dist-bin200.zip 18MB
jruby-dist-src.tar.gz 8MB
jruby-dist-src.zip 13MB

after unzipping jruby-dist-bin200.* archives you need to run bin/unpack200.sh
which takes a couple of seconds. did not yet tried for a windows unpack200.bat - first I need to know whether this is OK to go this route !

I personally would go for the bin200 archives - the reduction is really impressive and makes a difference in installing jruby (for me).

I also started to repack ruby-maven and ruby-maven-libs so they can be installed a default gems and refactored the script for installing default gems from lib/pom.rb into its own ruby file (need to reuse it for maven/jruby-dist) but this is still work in process.

I also added some length test in test/check_versions.sh which runs with travis to avoid the unwanted explosion of archive size we did see with jruby-9.0.0.0.pre1

@chrisseaton
Copy link
Contributor

I also added some length test in test/check_versions.sh which runs with travis to avoid the unwanted explosion of archive size we did see with jruby-9.0.0.0.pre1

That's a really good idea.

@headius
Copy link
Member Author

headius commented Mar 3, 2015

Very nice @mkristian, and I agree the size reduction is significant. Only thing that gives me pause is releasing too many distribution archives since I feel like we'd need to still provide the unpacked versions too.

What do you think @enebo?

@mkristian
Copy link
Member

mkristian commented Mar 3, 2015 via email

@enebo
Copy link
Member

enebo commented Mar 3, 2015

The original problem was dist was too big and the answer is to continue disting the big file and add more dist files to the release? I don't think this is actually solving a problem. I also don't think most users know what pack200 is and so they won't download it because they won't think they can unpack it. pack200, to me, fits better into installers since it can be hidden from the person installing.

I always feel like the dissenting voice :) We release waaay too many files already. In another couple of years I will be uploading 1GB of diced permutations of our bits (let us not forget we are not even doing truffle dists yet).

@mkristian
Copy link
Member

mkristian commented Mar 3, 2015 via email

@mkristian
Copy link
Member

mkristian commented Mar 3, 2015 via email

@headius
Copy link
Member Author

headius commented Mar 16, 2015

@mkristian Yes, cherry-pick any size improvements other than pack200 stuff to master. Perhaps some day we'll explore pack200 distribution.

@mkristian
Copy link
Member

@headius the explosion of the size of most distribution artifacts for 9.0.0.0.pre1 came through the embedded jruby-complete-1.7.19 inside the jruby-readline setup. this I fixed a while ago when I released jruby-readline-1.0.gem to maven central.

beside this there are quite a few place to reduce amount of maven modules and attached artifacts but that is not part of the pack200 branch.

@mkristian
Copy link
Member

so I reworked the artifacts which gets "deployed" to maven central:

  • two less maven modules:
    • maven/jruby-noasm is gone
    • maven/jruby-stdlib is part of lib/
  • three less deployed artifacts:
    • jruby-noasm.jar
    • jruby-core-noasm.jar which is the jruby-core.jar now
    • jruby-core-complete.jar is gone

the jruby-core pom uses the dependency-reduced pom after shading asm and jnr-jffi (which uses asm).

this change reduced the upload on release about 20MB.

finally I added jruby-dist-bin200.tar.gz which is 17MB. here I also replaced the bin/jruby script with one which unpacks all the jars when used the first time before executing the jruby. any further bin/jruby execution works as usual - on linux ONLY or when there is a bash.

so overall it is a reduction of attached artifacts and total upload size :)

see more details on pack200 branch.

once maven-3.3.0 is released I will work on the ruby-maven-libs.gem and reduce the dist size even more.

@mkristian
Copy link
Member

just a follow up:

with the new ruby-maven and ruby-maven-libs gem it is finally possible to install those gems as default gems inside the jruby-dist files which will reduce the size by at least 8MB

@mkristian mkristian added this to the JRuby 9.0.0.0.rc1 milestone Jun 1, 2015
@enebo
Copy link
Member

enebo commented Jun 26, 2015

@mkristian are these new versions on master now as default gems? Can this be closed now?

@mkristian
Copy link
Member

@enebo the new versions are on master but not as default gems. give me the
weekend to finalize it. had already done something but those scripts are on
my old laptop which is not with me right now.

On Fri, Jun 26, 2015 at 5:34 PM, Thomas E Enebo notifications@github.com
wrote:

@mkristian https://github.com/mkristian are these new versions on
master now as default gems? Can this be closed now?


Reply to this email directly or view it on GitHub
#2487 (comment).

@enebo
Copy link
Member

enebo commented Jun 26, 2015

@mkristian okie dokie...just checking

@mkristian
Copy link
Member

done on both branches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants