Memory leak in JRuby 1.7.25 #4232
Comments
|
These instance counts do indeed look strange. What we really need to see is a heap dump, so we can find these leaking objects and figure out why they are not being collected. You can get a heap dump using jmap. Ideally, it would not be a 10GB dump...hopefully something smaller (like in the 1-2GB range) would still clearly show the leaking objects. I'll have a look at the socket subsystem in 1.7 to see if anything obvious shows up. |
|
Another thing that would obviously help is if you could try to bisect this problem to at least a specific JRuby version, if not a specific revision. I do not see anything obvious looking at socket logic yet. |
|
Attached is a heap.bin created via jhat against jruby-1.7.25 running the command line script above after joining is complete but before nil'ing the thread array. I have a 1.1G snapshot taken, using YourKit, against the running Tomcat instance before we pulled it out of service. It potentially has our customer data in it which presents some challenges. Would you be comfortable with signing an NDA? If that's not feasible, maybe we could arrange a time for a remote screen share, or if you know what you're looking for you could tell us what you need and we can go get it for you. |
|
(FWIW, this heap.bin was created on OS X El Capitan, not on AWS Linux) |
|
Hmm...got sidetracked and lost track of this one. Can we pick it up again? I assume it is still a problem for you with 1.7.26, but that would be good to confirm if you have not already. |
|
Thanks to 9.1.7.0, last week we were able to deploy 9K in production without having to revert back to the 1.7.x series as we had had to do with all the previous 9K releases we had tried. At this point I am no longer concerned about 1.7.x's behavior, so if you want to close this I'm okay with that. |
|
Ok, I'll close this. We've had no other reports of this kind of leak in 1.7.25, and not much to go on for leads, so let's hope you were just lucky :-) |
Environment
Provide at least:
Other relevant info you may wish to add:
Expected Behavior
Actual Behavior
as was this one:
I understand that the site where the OoME exception may not highly correlate to where the leak is happening, but this Rails up does very little else beside take incoming HTTPS requests, make outbound HTTPS requests, and relay the results, so it seems likely that the issue is somewhere near the SSL integration.
Trivia
I attempted to follow the suggestions for profiling JRuby with a trivial example:
jruby -J-Djruby.reify.classes=true -rhttparty -e 'puts "Sleeping while you get the pid"; sleep 30; puts "Game on"; threads = []; 1000.times { threads << Thread.new { HTTParty.get "https://example.com"; nil } }; puts "Joining"; threads.each { |t| t.join }; puts "Sleeping before niling threads"; sleep 30; threads = nil; puts "Sleeping again"; sleep 60'I looked for objects that appeared in multiples of the 1000 threads that I used on the command line above. On 1.7.25, there appear to be a bunch that persist as long as the threads that ran the request exist, and vanish when the the
threadsarray are nil'ed, suggesting that they are thread local:I can't reproduce similar instance counts on 1.7.19.
The text was updated successfully, but these errors were encountered: