Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unusually slow performance on "sig" benchmarks #2860

Closed
headius opened this issue Apr 21, 2015 · 4 comments
Closed

Unusually slow performance on "sig" benchmarks #2860

headius opened this issue Apr 21, 2015 · 4 comments
Labels
performance
Milestone
Invalid or Duplicate

Comments

@headius
Copy link
Member

headius commented Apr 21, 2015

The "sig" library provides type-checking for Ruby. The benchmarks reported in their README don't reflect the performance we'd expect to see on JRuby 9000.

I've filed janlelis/sig#1 to hopefully get to the bottom of it.
@headius headius added this to the 9.0.0.0.rc1 milestone Apr 21, 2015
@enebo
Copy link
Member

enebo commented Apr 21, 2015

A few observations after a casual glance:

  1. Module prepends a define_method (which ends up having us go through block for execution)
  2. this method calls super and I believe as a proc we end up fallng back to unresolved super
  3. The method accepts **kwrest. Our kwarg impl is pretty slow atm

The first two we will eventually be able to eliminate in post 9k work since we plan on converting define_method procs to actual methods. This will also allow us to make the unresolved super resolved. The kwargs overhead I have no doubt we will be improving a bit early in 9k cycle.

@headius
Copy link
Member Author

headius commented Apr 27, 2015

I updated the referenced issue with results from running the benchmark directly. The original results were run via a Rakefile, so the block-based IPS bench never jitted. Our performance is orders of magnitude better than what is in the README if JIT runs with indy enabled.

Our performance on the sig and contracts part of the bench now is closer to where it should be, but not where I'd like to see it:

[] ~/projects/sig $ rvm ruby-2.2 do ruby -I lib blah.rb 
ruby version: 2.2.0
sig version: 1.0.1
contracts version: 0.9
Calculating -------------------------------------
                pure    85.808k i/100ms
                 sig    11.573k i/100ms
           contracts     6.723k i/100ms
-------------------------------------------------
                pure      6.281M (±11.6%) i/s -     30.977M
                 sig    163.279k (± 6.0%) i/s -    821.683k
           contracts     83.552k (± 5.4%) i/s -    416.826k

Comparison:
                pure:  6281161.5 i/s
                 sig:   163278.7 i/s - 38.47x slower
           contracts:    83552.4 i/s - 75.18x slower


[] ~/projects/sig $ jruby -Xcompile.invokedynamic=true -Ilib blah.rb
ruby version: 2.2.2
sig version: 1.0.1
contracts version: 0.9
Calculating -------------------------------------
                pure    99.780k i/100ms
                 sig    11.177k i/100ms
           contracts     1.528k i/100ms
-------------------------------------------------
                pure     46.102M (±18.4%) i/s -    215.724M
                 sig    186.208k (± 5.5%) i/s -    938.868k
           contracts     51.995k (± 8.1%) i/s -    258.232k

Comparison:
                pure: 46101534.6 i/s
                 sig:   186207.7 i/s - 247.58x slower
           contracts:    51994.7 i/s - 886.66x slower

The sig library uses prepend, so there's no way to compare with JRuby 1.7.

@headius
Copy link
Member Author

headius commented Apr 27, 2015

I realized that the type-checking wrapper sig generates uses define_method, and so it will never JIT in the normal situation (yet). Results with -Xjit.threshold=0 are better:

[] ~/projects/sig $ jruby -Xcompile.invokedynamic=true -Xjit.threshold=0 -Ilib blah.rb
ruby version: 2.2.2
sig version: 1.0.1
contracts version: 0.9
Calculating -------------------------------------
                pure   200.902k i/100ms
                 sig    20.817k i/100ms
           contracts     4.867k i/100ms
-------------------------------------------------
                pure     31.938M (±15.7%) i/s -    152.686M
                 sig    386.760k (± 6.4%) i/s -      1.936M
           contracts    125.764k (± 8.1%) i/s -    627.843k

Comparison:
                pure: 31937956.7 i/s
                 sig:   386760.3 i/s - 82.58x slower
           contracts:   125764.4 i/s - 253.95x slower

All numbers are comfortably ahead of MRI here.

The numbers for rbx are very confusing, though. They show a significant drop in perf for contracts but nearly identical performance between the pure and sig benchmarks. That doesn't seem possible; there's a lot of overhead in the type checking logic.

@headius
Copy link
Member Author

headius commented Apr 28, 2015

This is no longer an issue; with proper configs/flags we are easily the fastest at these benchmarks.

The rbx numbers turned out to be bogus because they have a JIT bug preventing sig's typechecking from running in the jitted version of those methods.

@janlelis If you want to get accurate JRuby results, try running with -Xcompile.invokedynamic=true -Xjit.threshold=0 to use invokedynamic and compile everything right away. I think the rbx results should just be removed, since they aren't accurate at all.

@headius headius closed this as completed Apr 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
Invalid or Duplicate
Development

No branches or pull requests
2 participants
@headius @enebo