Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible memory leak in 1.7.22 #3339

Closed
TheSynthMaster opened this issue Sep 21, 2015 · 10 comments
Closed

Possible memory leak in 1.7.22 #3339

TheSynthMaster opened this issue Sep 21, 2015 · 10 comments
Labels
JRuby 1.7.x windows
Milestone
JRuby 1.7.23

Comments

@TheSynthMaster
Copy link

I have a web app which was on jruby 1.7.9 (1.9.3p392) 2013-12-06 87b108a on Java HotSpot(TM) 64-Bit Server VM 1.8.0_40-b26 [Windows Server 2008 R2-amd64] on one instance and jruby 1.7.9 (1.9.3p392) 2013-12-06 87b108a on Java HotSpot(TM) Client VM 1.7.0_0
7-b10 [Windows 2003-x86] on another instance with no problems

Now I have updated to jruby 1.7.22 (1.9.3p551) 2015-08-20 c28f492 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 +jit [Windows Server 2008 R2-amd64] on one instance and jruby 1.7.22 (1.9.3p551) 2015-08-20 c28f492 on Java HotSpot(TM) Client VM 1.8.0_60-b27 +jit [Windows 2003-x86] on the other

I've transitioned from rails 4.1.0 to 4.2.4

I'm experiencing constant jvm crashes on both instance after few hours under little to moderate load due to java heap full... I tried increasing the heap to 2g but it only takes a little more to fill...

Tried several jvm params for Xmn,Xms,Xmx and 3 different garbage collectors (auto, Concurrent Mark and Sweep (CMS) and the new G1) with no luck

In the transition I also updated all the gems, both rails and others gems

Previous gem list

*** LOCAL GEMS ***

actionmailer (4.1.0)
actionpack (4.1.0)
actionview (4.1.0)
activemodel (4.1.0)
activerecord (4.1.0)
activerecord-deprecated_finders (1.0.3)
activerecord-oracle_enhanced-adapter (1.5.4)
activerecord-session_store (0.1.0)
activesupport (4.1.0)
afm (0.2.0)
arel (5.0.1.20140414130214)
Ascii85 (1.0.2)
atomic (1.1.16 java)
builder (3.2.2)
bundler (1.6.2)
coffee-rails (4.0.1)
coffee-script (2.2.0)
coffee-script-source (1.7.0)
daemons (1.1.9)
erubis (2.7.0)
eventmachine (1.0.3 java)
execjs (2.0.2)
hashery (2.1.1)
hike (2.1.3, 2.0.0, 1.2.3)
i18n (0.6.9)
jquery-rails (3.1.0)
jruby-win32ole (0.8.5)
json (1.8.1 java)
looking_for (0.3.0)
mail (2.5.4)
mime-types (2.2, 1.25.1)
minitest (5.3.3)
multi_json (1.10.0, 1.9.2)
pdf-core (0.2.5)
pdf-reader (1.3.3)
polyglot (0.3.4)
prawn (1.0.0)
puma (2.8.2 java)
rack (1.5.2)
rack-test (0.6.2)
rails (4.1.0)
railties (4.1.0)
rake (10.3.1, 10.3.0, 10.1.1)
ruby-rc4 (0.1.5)
rubygems-update (2.3.0, 2.2.2)
sass (3.3.7, 3.3.5, 3.2.13)
sass-rails (4.0.3)
sprockets (2.12.1, 2.12.0, 2.10.1)
sprockets-rails (2.1.3)
thor (0.19.1)
thread_safe (0.3.3 java)
tilt (2.0.1, 1.4.1)
treetop (1.5.3, 1.4.15)
ttfunk (1.1.1)
tzinfo (1.2.2)
tzinfo-data (1.2014.9)
uglifier (2.5.0)
will_paginate (3.0.5)

current gem list

*** LOCAL GEMS ***

actionmailer (4.2.4)
actionpack (4.2.4)
actionview (4.2.4)
activejob (4.2.4)
activemodel (4.2.4)
activerecord (4.2.4)
activerecord-oracle_enhanced-adapter (1.6.3)
activerecord-session_store (0.1.1)
activesupport (4.2.4)
arel (6.0.3)
builder (3.2.2)
bundle (0.0.1)
bundler (1.10.6)
coffee-rails (4.1.0)
coffee-script (2.4.1)
coffee-script-source (1.9.1.1)
erubis (2.7.0)
execjs (2.6.0)
globalid (0.3.6)
i18n (0.7.0)
jar-dependencies (0.1.15)
jquery-rails (4.0.5)
jruby-openssl (0.9.10 java)
json (1.8.3 java, 1.8.0 java)
loofah (2.0.3)
looking_for (0.3.0)
mail (2.6.3)
mime-types (2.6.2)
minitest (5.8.0)
nokogiri (1.6.6.2 java)
pdf-core (0.4.0)
prawn (1.3.0)
prawn-table (0.2.2)
puma (2.13.4 java)
rack (1.6.4)
rack-test (0.6.3)
rails (4.2.4)
rails-deprecated_sanitizer (1.0.3)
rails-dom-testing (1.0.7)
rails-html-sanitizer (1.0.2)
railties (4.2.4)
rake (10.4.2, 10.1.0)
rdoc (4.1.2)
ruby-maven (3.3.3)
ruby-maven-libs (3.3.3)
sass (3.4.18)
sass-rails (5.0.4)
sprockets (3.3.4)
sprockets-rails (2.3.3)
thor (0.19.1)
thread_safe (0.3.5 java)
tilt (2.0.1)
ttfunk (1.4.0)
tzinfo (1.2.2)
tzinfo-data (1.2015.6)
uglifier (2.7.2)
will_paginate (3.0.7)
@TheSynthMaster
Copy link
Author

I'm not sure it's jruby to blame, maybe it's rails or some gems... but the problem definitely exists... I'm doing some tests limiting puma to 2 threads from the standard 16 it can spawn by default, and it seems to give some benefits, but I really need to test more...

P.S. limiting puma to 2 threads do not solve the problem, maybe it postpone it a little...

@TheSynthMaster
Copy link
Author

Eg. with JAVA_OPTS=-Xmx1536m -Xms768m

with the old version after 3 days the jvm uses 800 megs, new version after 1 hour is already using 1.7 Gb... (and with puma limited to only 2 threads...)

@rtyler
Copy link

rtyler commented Sep 21, 2015

@PanDar1 is it possible for you to profile the running JVM to understand more about where the memory usage is coming from? I find VisualVM to be very helpful for this.

@rtyler rtyler added the windows label Sep 21, 2015
@TheSynthMaster
Copy link
Author

Ok, I run two sessions with the load generate from only one client, this is the profiler output
(please change extension to .nps for the snap and .rar for the profiler out), both sessions crashed after less than 10 minutes under light load

jvm_crash_snap

jvm_crash

@TheSynthMaster
Copy link
Author

Any thoughts? Even limiting jruby to one instance with -Djruby.max.runtimes=1 did not solved the problem...

@TheSynthMaster
Copy link
Author

Any progress? I'm still waiting, but if the problem will not be solved I will have to leave jruby and turn back to mri, this problem is too big for a production environment...

@enebo enebo added this to the JRuby 1.7.23 milestone Oct 15, 2015
@enebo enebo modified the milestones: JRuby 1.7.23, JRuby 1.7.24 Nov 24, 2015
@TheSynthMaster
Copy link
Author

Installed in devel environment 1.7.23, from first tests it seems to work, I will report stress test on production later... Tnx

@mkristian
Copy link
Member

@PanDar1 did the gem list change again, after switching to 1.7.23 ?

@TheSynthMaster
Copy link
Author

Yes, some gems has been updated, every time I change the jruby version I have to reinstall all the gems... this are the changed ones:

actionmailer (4.2.5)
actionpack (4.2.5)
actionview (4.2.5)
activejob (4.2.5)
activemodel (4.2.5)
activerecord (4.2.5)
activerecord-oracle_enhanced-adapter (1.6.5)
activerecord-session_store (0.1.2)
activesupport (4.2.5)
coffee-script-source (1.10.0)
jar-dependencies (0.2.6)
jruby-openssl (0.9.11 java)
jruby-win32ole (0.8.5)
mime-types (2.99)
minitest (5.8.3)
nokogiri (1.6.7 java)
puma (2.15.3 java)
rails (4.2.5)
railties (4.2.5)
sass (3.4.19)
sprockets (3.4.1)
tzinfo-data (1.2015.7)

@headius
Copy link
Member

headius commented Jan 15, 2016

@PanDar1 Have you been able to re-test with 1.7.23? I'm going to close this bug based on your initial assessment that things are working. If you have further issues, let us know.

@headius headius closed this as completed Jan 15, 2016
@headius headius modified the milestones: JRuby 1.7.23, JRuby 1.7.24 Jan 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
JRuby 1.7.x windows
Projects
None yet
Milestone
JRuby 1.7.23
Development

No branches or pull requests
6 participants
@headius @enebo @rtyler @mkristian @kares @TheSynthMaster