maybe try not running benchmarks in irb next time - require should have no effect when running jruby

@kares please see the linked git repo for reproducing the results from a rake task.

yeah ... just try running ruby fib.rb straight I do not see issues. your repo does bench as part of rake which I'm not sure is a good idea either. usual thing is to fork another ruby process.

@kares @creddy with non-indy support on I do definitely see fib running a bit slower on 9.0.5.0 (MacOS + Java 8u60) than 1.7.25 but I don't find that very important. Also I notice enabling invokedynamic and we get a big drop in perf. So we might want to do some sanity checking on the perf difference in this version of fib...but like I said the numbers are not way off...

Why I am really reopening this issue is the massive drop in performance in loading arjdbc gem. What the hell is going on?

w/o require:

jruby ../snippets/bench4.rb 
Warming up --------------------------------------
              fib up   476.000  i/100ms
Calculating -------------------------------------
              fib up      4.687k (± 4.1%) i/s -    140.420k

w/ require

jruby ../snippets/bench4.rb 
Warming up --------------------------------------
              fib up   243.000  i/100ms
Calculating -------------------------------------
              fib up      2.494k (± 4.0%) i/s -     74.844k

The performance drops in half by loading this library? @kares does activerecord-jdbcsqlite3-adapter monkey patch any core classes? I would not think it could have any effect on this tiny fib method. This is something we should be able to explain.

Oh, I bet it is monkey patches like days and hours. We see that Fixnum has been modified and revert to slow path math.

hmm, wasn't seeing any notable difference as I tried. but you guys are right that require -> require 'arjdbc' -> require 'active_record' -> require 'active_support' parts (does core patches)

turns out was running this from under arjdbc repo where I loaded a stub file thus not requiring anything!

@kares aha...thanks for pointing out ar needs as.

@headius we don't keep opted math once we add a new method to the type because any possible call to that type may change fixnum's state internally right? This makes me wonder if we can leverage IR to know whether we are changing internal state. If we do not putfield in the added/replaced method then it can be added to the type and we know normal math still applies. I guess this is an additional burden on us checking to see if the type changed but I can see it as a path to not having this limitation...

@headius ignore my last question. This is just a simple type guard which is getting set up before AS is loaded since it is the script file. My question about internal state also makes no sense for an immediate type so that would perhaps be a question for a non-immediate type if it ever arises (even then putfield is inadequate due to metaprogramming or eval).

@enebo, could you clarify your comment regarding the relative performance of 1.7 vs 9.0? You said:

with non-indy support on I do definitely see fib running a bit slower on 9.0.5.0 (MacOS + Java 8u60) than 1.7.25 but I don't find that very important

I was hoping that you would find the performance difference to be important. The reason that I have not moved to 9.0 is because we see a significant performance degradation (30% slower) compared to the same app running on 1.7, much like the OP stated, and is in-line with 33% performance degradation in the fib test that the OP presented.

I had previously talked about this performance issue in #3595: The spec performance turned out to be a red hearing, but the rails performance is still there, I just had not re-open the issue.

@ylansegal yeah I am extremely interested in why Rails is not performing well. Fib benchmark, not as much for two reasons:

1. fib is pretty atypical as a micro benchmark for Ruby. It is highly recursive and mostly primitive math. It is also probably the most understood benchmark we have. Odds are whatever issue we find wrong with Rails performance will not be related to recursive math-heavy functions.
2. we have plans in the next year to enable numeric unboxing as a profiled optimization. This will end up giving a big bump to fib since most of the overhead is in our boxing of long into a RubyFixnum instance.

More generally though when we put out 9k we knew some stuff was a bit slower and others were a bit faster. Our groundwork for larger optimizations (like unboxing mentioned above) will end up putting us into a new realm of performance.

More specifically to the reported issue that Rails is not giving the performance relative to 1.7 we need to get some better visibility on why this is. We have gotten several reports of things being slower and some on us being faster on 9k. I suspect there will be some common thread which shows us doing something poorly on 9k that we do not realize yet.

Thanks @enebo for the explanation. As time permits, I'll try to do some more investigation about which parts slow down our app and report if I find something interesting.

@enebo and @headius Thank you for taking an interest in this. It is very much appreciated.

@kares You mentioned

usual thing is to fork another ruby process.

For when I run benchmarks in the future, would you mind explaining how one would accomplish this in JRuby/JVM? If you have time a pull request against https://github.com/creddy/jruby-bench would be even better. Thank you.

It seems appropriate to add our own findings regarding the jruby 9k performance problems.

We recently tried upgrading one of our core services from 1.7.15 to 9.0.4.0 and saw a similar 20-30% drop in performance. With invokedynamic enabled it was even worse.

Some details:

• sinatra app running with trinidad processing between 4k-10k rpm per server (of which we have currently 16)
• JVM
  • oracle v1.8.0_72 with -XX:+UseG1GC
• average response time
  • 1.7.15 ~40ms
  • 9.0.4.0 50-55ms
  • 9.0.4.0 with indy 55-70ms
• we never tried 1.7 with indy. We might do that now after we moved to the latest 1.7 release.
• among lots of other gems we use active_record(via the jdbc-adapter) and active_support (this seem to be a common theme with the slowdowns) I can provide a Gemfile.lock for a complete list of gems
• with invoke dynamic enabled it took at least 45 minutes to get consistent response times which where even worse than without invoke dynamic
• I've done quite a bit of profiling of a fully warmed up JVM. I couldn't find a big bottleneck in our code. "Everything" seems to be a bit slower
• interestingly normal development and running specs seems quite a bit faster using 9k which leads me to believe the 1.7 jit is working at the moment better than the new one.

Hi,
I have a performance problem with JRuby 9.0.5.0, too.

I observed the following:

A JRuby program needs 1 minute to load all gems on my laptop.
If I disconnect my laptop from the network the same program needs only 8 seconds.
After some debugging I found out that

uri:classloader://specifications/asciidoctor-1.5.4.gemspec

yields to a

new File("//specifications/asciidoctor-1.5.4.gemspec").exists()

Under Windows "//specifications" is a network drive, which can not be resolved.
I think this is the reason for my performance problem in case the laptop is connected to the network.

Best regards
Kisho

@kisho23 najs find, please open a new issue instead of commenting on an unrelated one and if possible include more details. a trace of where that new File("//specifications/asciidoctor-1.5.4.gemspec").exists() is coming from (or how does someone reproduce). thanks!

@kisho23 Yeah I concur with @kares...that is very interesting! Please open a new issue of your own!

Done:
#3744

Those of you who have noticed performance issues, please try 9.1 snapshots from http://ci.jruby.org. Many performance bottlenecks have been fixed.

Since we can reproduce this one, I'll poke at it a bit more today.

Confirmed with 9.1 this perf difference is still there. If my theory is correct we should be able to simulate it by simply modifying Fixnum.

Ok, unexpectedly, turning off the guard for reopened Fixnum does not appear to make performance return. Continuing to dig.

Ok, here's my findings. I had not noticed the unusual condition (creating a new Range and calling Range#include?) in the benchmark, which created a lot of object overhead and confused me a bit.

First off, performance with or without Fixnum reopening guards was about the same as you report, with 9k w/ require performing half as well as 9k w/o require.

9k w/o require on original bench:

jruby -Xcompile.invokedynamic fib_bench.rb 
Warming up --------------------------------------
              fib up   894.000  i/100ms
Calculating -------------------------------------
              fib up      9.155k (± 5.7%) i/s -    273.564k

9k w/ require on original bench

Warming up --------------------------------------
              fib up   367.000  i/100ms
Calculating -------------------------------------
              fib up      3.831k (± 4.8%) i/s -    114.871k

If I modify the benchmark to use < 2 rather than the Range#include? call, however, things look very different:

JRuby w/o require:

Warming up --------------------------------------
              fib up   145.428k i/100ms
Calculating -------------------------------------
              fib up     20.510M (±13.6%) i/s -    598.727M

JRuby w/ require

Warming up --------------------------------------
              fib up   119.957k i/100ms
Calculating -------------------------------------
              fib up     19.632M (±13.7%) i/s -    572.315M

JRuby w/ require but Fixnum reopening guards disabled:

[] ~/projects/jruby $ jruby -Xcompile.invokedynamic -r active_record -r activerecord-jdbcsqlite3-adapter fib_bench.rb 
Warming up --------------------------------------
              fib up   142.976k i/100ms
Calculating -------------------------------------
              fib up     19.890M (±13.4%) i/s -    580.626M

This tells me that the Fixnum reopening is likely only having a small impact here, but that Range#include? bit seems to be much slower when Rails is loaded.

Ok...I think this bug is a red herring. Rails monkey-patches Range#include with a pure-Ruby version: https://github.com/rails/rails/blob/4-2-stable/activesupport/lib/active_support/core_ext/range/include_range.rb

Note that this version delegates to the original version when not testing include? against another Range, but this extra logic is the reason why your fib implementation slowed down under Rails.

I modified the bench to use Range#cover? and it does not degrade. Obviously the better change is to not use a Range for the condition.

FWIW, MRI degrades on the original benchmark as well, but not by as much since they're already quite a bit slower.

Ruby 2.3 w/o require

Warming up --------------------------------------
              fib up   301.000  i/100ms
Calculating -------------------------------------
              fib up      3.004k (± 5.6%) i/s -     89.999k in  30.059173s

Ruby 2.3 w/ require

Warming up --------------------------------------
              fib up   204.000  i/100ms
Calculating -------------------------------------
              fib up      2.029k (± 4.5%) i/s -     60.792k in  30.017293s

I'm going to close this since there's really nothing to fix. Might open a bug to track improvements to our Fixnum/Float reopening logic.

Thank you for taking the time to investigate 👍