Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Truffle] Use Array#{max,min} from rubinius #3322

Closed
wants to merge 1 commit into from

Conversation

brauliobo
Copy link

Java's comparable works quite differently from the <=> ruby operator

Benchmark:

$: << '/home/braulio/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/benchmark-ips-2.3.0/lib'
require 'benchmark/ips'
a = [5,4,10,12,23,3,4]
Benchmark.ips do |x|
 x.config time: 5, warmup: 30
 x.report{ |t| t.times{ a.max } } 
end

Rubinius version:

Calculating -------------------------------------
                    compilation failed!?
[truffle] opt fail         now:time.rb:28 <split-12-U>                                 |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
   502,000  i/100ms
-------------------------------------------------
                    compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
     10,671k (± 20,7%) i/s -     50,200k

Original version:

Calculating -------------------------------------
                    compilation failed!?
[truffle] opt fail         match_io:slex.rb:203                                        |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         match_io:slex.rb:203                                        |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         now:time.rb:28 <split-0-U>                                  |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
compilation failed!?
[truffle] opt fail         job.rb:235<OSR>                                             |Reason java.lang.StackOverflowError 
   403,000  i/100ms
-------------------------------------------------
                         10,568k (± 18,7%) i/s -     49,972k

@brauliobo
Copy link
Author

I have used git-new-workdir for using both branches simultaneously. I think it is ok as they will have different target folders, right?

@chrisseaton chrisseaton self-assigned this Sep 13, 2015
@chrisseaton chrisseaton added this to the truffle-dev milestone Sep 13, 2015
@chrisseaton
Copy link
Contributor

I'm waiting on some other changes to be checked carefully for performance regression before I try applying this. Running all of our benchmarks takes a while.

@brauliobo
Copy link
Author

@chrisseaton i'll try this on truffle-head as soon as possible, maybe the errors are different there.

@eregon
Copy link
Member

eregon commented Sep 16, 2015

block_given? is likely a killer here too.

@brauliobo
Copy link
Author

@eregon yeah, pretty poor implementation, it calls block_given? for each element.

but yet we need a proof that it is slower

  def max
    max = undefined
    each do
      o = Rubinius.single_block_arg
      if undefined.equal? max 
        max = o 
      else
        comp = block_given? ? yield(o, max) : o <=> max 
        unless comp
          raise ArgumentError, "comparison of #{o.class} with #{max} failed"
        end 

        if Comparable.compare_int(comp) > 0 
          max = o 
        end 
      end 
    end 

    undefined.equal?(max) ? nil : max 
  end 

@eregon
Copy link
Member

eregon commented Sep 16, 2015

@brauliobo the real problem is that block_given? needs to get the caller frame currently to know if that frame has a block, and that iterates the stack and deoptimizes, which is not fast.
We should be able to move the check out of the loop or at least make it very cheap once we have fast access to the caller frame.

@thedarkone
Copy link
Contributor

(off topic to this PR)

@eregon is it forever game over for methods containing block_given?, or is this a temporary truffle/graal/jruby deficiency?

@chrisseaton
Copy link
Contributor

We have an optimisation in mind to fix block_given? by supplying a direct reference to the caller frame when calling some methods, so one does not have to be found by walking the stack. This will allow references to local variables and things like the block in the caller frame to be constant folded, which is what block_given? needs. This will also allow us to do some mind-bending constant folding such as binding_of_caller.local_variable_get(:x) * y.

The optimisation is simple but will require a bit of refactoring. We should get around to it.

@chrisseaton
Copy link
Contributor

block_given? was fixed. It might be time to try this again. @brauliobo would you like me to show you how to test if this now performs as well without the Java code? Or if you aren't interested in that I'll do it myself.

@brauliobo
Copy link
Author

@chrisseaton will do some benchmarks asap

@chrisseaton
Copy link
Contributor

@brauliobo do you want us to take another look at this PR? If you update it to latest JRuby we'll help you run the benchmarks and see if it needs any more work before it can be merged.

@brauliobo
Copy link
Author

yes @chrisseaton please, really would like to look into this into this but time is running away. thank you :)

@chrisseaton
Copy link
Contributor

Sorry, not sure what you mean. Do you mean you don't have time at the moment, but will do soon? Or you're unlikely to have time to do this any time soon so we should do it?

@brauliobo
Copy link
Author

it is better that you do it because being realistic I won't have enough time for this.
thanks for the good job with jruby truffle :)

@brauliobo
Copy link
Author

brauliobo commented Apr 29, 2016

~/P/j/j/master git:master ❯❯❯ JAVACMD=~/Projects/jruby/jvmci/jdk1.8.0_92/product/bin/java bin/jruby -X+T ../bench.rb
Warming up --------------------------------------
                         2,209M i/100ms
Calculating -------------------------------------
                         58,790M (± 18,2%) i/s -    229,747M in   5,009333s
~/P/j/j/max git:test-array-max-min ❯❯❯ JAVACMD=~/Projects/jruby/jvmci/jdk1.8.0_92/product/bin/java bin/jruby -X+T ../bench.rb
Warming up --------------------------------------
                         2,077M i/100ms
Calculating -------------------------------------
                         52,888M (± 18,5%) i/s -    207,664M in   5,002687s

@chrisseaton
Copy link
Contributor

Looks great! I'm just setting up a new internal CI and benchmarking system, which I'd like to run this on to give us some more confidence, so I'm sorry I'll just need to delay you by a week or so before we can merge this.

@chrisseaton
Copy link
Contributor

I'm also going to set up some things called PE tests - I'll get back to you about those.

@brauliobo
Copy link
Author

benchmark used:

$: << '/home/braulio/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/benchmark-ips-2.6.1/lib'                                                                                                 
require 'benchmark/ips'                                                                                                                                                                      
a = [5,4,10,12,23,3,4]                                                                                                                                                                       
Benchmark.ips do |x|                                                                                                                                                                         
  x.config time: 5, warmup: 30                                                                                                                                                               
  x.report{ |t| t.times{ a.max } }                                                                                                                                                           
end                                                                                                                                                                                          

@brauliobo brauliobo force-pushed the test-array-max-min branch 2 times, most recently from 8c2ef65 to 6c61494 Compare April 29, 2016 19:21
@brauliobo
Copy link
Author

@chrisseaton updated the benchmarks after doing two separate builds for each branch using git-new-workdir to ensure everything is correct. now the ruby version appears to be a bit slower

@chrisseaton
Copy link
Contributor

We should revisit now our benchmarks are stable again after some changes to Graal.

Java's comparable works quite different from the <=> ruby operator
@brauliobo
Copy link
Author

brauliobo commented Sep 10, 2016

master version:

~/P/j/j/master git:master ❯❯❯ JAVACMD=~/Projects/jruby/graal/graalvm-0.16-dev-dk/bin/java bin/jruby -X+T ../bench.rb
Warming up --------------------------------------
                    2068814,000M i/100ms
Calculating -------------------------------------
                     8843082613733733,000M (±1289068246665058,0%) i/s - 386868218,000M in 5010767353050232,000000s

ruby version:

~/P/j/j/max git:test-array-max-min ❯❯❯ JAVACMD=~/Projects/jruby/graal/graalvm-0.16-dev-dk/bin/java bin/jruby -X+T ../bench.rb
Warming up --------------------------------------
                    2032078,000M i/100ms
Calculating -------------------------------------
                     8675873750696292,000M (±12919911379646804,0%) i/s - 390158976,000M in 5016019309898376,000000s

Performance is roughly the same with constants.

@chrisseaton
Copy link
Contributor

What's going on with those numbers? 8675873750696292,000M, ±12919911379646804,0%, 5016019309898376,000000s.

@chrisseaton
Copy link
Contributor

Also, I think ironically MRI now has separate methods for Array#min and Enumerable#min, but that's a problem we can solve later if we merge this now.

@brauliobo
Copy link
Author

Using a slightly different script of a randomly generated array:

$: << '/home/braulio/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/benchmark-ips-2.6.1/lib'
require 'benchmark/ips'
a = 10.times.map{ rand(1..20) }
Benchmark.ips do |x|
  x.config time: 5, warmup: 30
  x.report{ |t| t.times{ a.max } }
end

Makes the java version perform a bit faster:

~/P/j/j/master git:master ❯❯❯ JAVACMD=~/Projects/jruby/graal/graalvm-0.16-dev-dk/bin/java bin/jruby -X+T ../bench-rand.rb                                                                  ◼
Warming up --------------------------------------
                    2005529,000M i/100ms
Calculating -------------------------------------
                     8165281379313222,000M (±15329656650549875,0%) i/s - 346956517,000M in 5012463420974732,000000s
~/P/j/j/max git:test-array-max-min ❯❯❯ JAVACMD=~/Projects/jruby/graal/graalvm-0.16-dev-dk/bin/java bin/jruby -X+T ../bench-rand.rb
Warming up --------------------------------------
                    197074,000M i/100ms
Calculating -------------------------------------
                     7037981787634672,000M (±1561739903804728,0%) i/s - 30152322,000M in 5004666628997803,000000s

@brauliobo
Copy link
Author

brauliobo commented Sep 10, 2016

@chrisseaton good question about the numbers, do you recommend something besides benchmark-ips?

@brauliobo
Copy link
Author

upgraded to benchmark-ips 2.7.2 and still get the same weird numbers

@chrisseaton
Copy link
Contributor

benchmark-ips is excellent - but something is clearly wrong with 5012463420974732,000000s - that's 158 thousand milena of benchmarking time you've done there!

@chrisseaton
Copy link
Contributor

I'll pull locally and try tomorrow, and I'll run against out internal CI system as well. I'll update tomorrow.

@chrisseaton
Copy link
Contributor

chrisseaton commented Oct 5, 2016

It looks like this applied cleanly this time and didn't have any impact on performance. Sorry it took a year to get that in! We had the issue with block_given? which required some innovation, and then our benchmarks were a bit all over the place for a while so we couldn't see if it was safe to apply. Thanks!

@chrisseaton chrisseaton closed this Oct 5, 2016
@chrisseaton
Copy link
Contributor

The final commit applying it is d7ca7b7

@brauliobo
Copy link
Author

nice @chrisseaton, no worries and thanks!

@brauliobo brauliobo deleted the test-array-max-min branch October 6, 2016 21:22
@enebo enebo added this to the Non-Release milestone Dec 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants