Very high load in 1.9 mode #1713
Comments
|
Sorry nobody has gotten back to you. Do you still have this problem with later versions of JRuby? I would expect that 1.9 mode would have higher load than 1.8, as with things like encoding the language is simply more complicated. JRuby uses the same optimisations for both 1.8 and 1.9, so there aren't extra optimisations to compensation, like there are in MRI 1.8 to 1.9. |
|
Hi Chris, Thank you for the response. I deployed again my app, this time using the latest JRuby (1.7.16) to a server with a very low load (very few users use this machine). Unfortunately, the problem is still there. And it looks like it's not related to just higher load related to more complicated String operations, and other Ruby 1.9 features. It's working fine, and then, after some time suddenly it's starting using 100% of one CPU core, and than in random point of times it's using even more cores, and it never ends. Below are attached two images. One shows CPU usage from the last 24 hours after I switched to the latest JRuby. The other shows load from the last month and you can see what I described above (more cores getting used, until the app gets restarted). There is no such problem at all with 1.8 mode.
|
|
Just checking I'm reading the graph right - you switch from 1.8 to 1.9 mode in the middle of the 24 graph you just posted? |
|
No, I just updated JRuby to the latest version before the 24 hours graph starts, so the load was low, because app was freshly restarted. Both graphs show load in 1.9 mode. As you can see the problem is that suddenly, out of nowhere, after some time, load surges and JRuby uses 100% of CPU (one core). After more time also it may start using further cores (1 month graph). Nothing like this happens when the app uses 1.8 mode. |
|
What size heap are you running with? Can you try doubling it and see if that makes a difference? Eg -J-Xmx2G. See if it takes longer to go crazy, if nothing else. Maybe it spends all of its time in GC after a certain point - that would explain using all the cores. Sent from my iPhone
|
|
@ocher I would also examine a few heap dumps to see if something obvious is being executed on running threads. If it is callMethod or something like that it is Ruby code perhaps going in a tight loop. If you see some nio select logic then some internal IO logic is going crazy. Even posting a typical set of stacks from a dump might help shed some light on this. |
|
@chrisseaton I think that it's not a problem with GC. I've got a full GC log (you can check it yourself here: https://www.dropbox.com/s/b9oz1jyyzxa5nof/kirk_gc_jruby_1-7-16.log.gz?dl=0) and there aren't too many full GCs performed. @enebo For now I can provide you -Xrunhprof:cpu=samples log file. As you will see in the file, there are many threads created by: site_target_ranking.rb:42 file. This line invokes HTTPClient#delete method. I had to change HTTPClient because of this bug: #1891, but as far as I remember it shouldn't be a source of the problem described in the current issue, because the high load was there even before I patched HTTPClient (here is the patch I'm using: https://gist.github.com/ocher/750f5628241dcf25a6fe) |
|
interesting piece esp. since there's smt of the same at elastic/logstash#3989 "resolved" by avoiding wonder if it would manifest the same under JRuby 9K ... or if @ocher managed to get it working with 1.7.x ? |
|
no feedback in a few months. if anyone lands here please try to provide detailed input against JRuby 9K |
After porting my Rails app to 1.9 mode I noticed a very high load compared to 1.8 mode.
It's not a GC problem (we were talking about that on #jruby). Also JRuby 1.7.11 and 1.7.4 versions were checked (1.7.4 version was checked because of more complex encoding code in 1.7.11), and both have the same problem in 1.9 mode.
Also, as you can see on the attached load chart, there is a place (compared to the previous day) when load should go down, but still there is something running which uses a lot of CPU.
CPU samples (-Xrunhprof:cpu=samples):
The text was updated successfully, but these errors were encountered: