Can you provide the error as well? I don't know why there would be an issue patching an extension class, though frequently extensions make direct calls to the Java methods (as do C extensions in some cases), bypassing your patches.

Well I've done a bit more research trying to produce a simple test case (but that doesn't produce the error), so it may have something to do with where the patch is done

This was the error I got:-

Java::OrgJrubyExceptions::RaiseException
  (ArgumentError) wrong number of arguments calling `initialize` (3 for 0)
    RUBY.mouse_moved(mesh_doodle.rb:62)

So it looks like the Vec3D of the patch is seen in this case (lacks constructor of the original)?

PS: I was using java 8 lambda in constructor but I think that is irrelevant as I used the same in my simple test case, which failed to create error. This is probably an edge case, and there is a decent workaround, do you think it needs pursuing?

Had a bit of brain-wave, created and this test case https://github.com/ruby-processing/monkey reproduces behaviour, whether it is an error or just something to advise about...
rake to compile and run tests.

@headius Updated build to use polyglot build https://github.com/ruby-processing/monkey, problem seem to be that the regular monkey patching does something to class constructor.
https://travis-ci.org/ruby-processing/monkey/builds/101874806

Just thought I had better test using without lambda8 constructor:-

  /**
  *
  * @param runtime
  */
  public static void createVec(final Ruby runtime) {
      RubyClass vecCls = runtime.defineClass("Vec2D", runtime.getObject(), new ObjectAllocator() {
        @Override
        public IRubyObject allocate(Ruby runtime, RubyClass rubyClass) {
          return new Vec(runtime, rubyClass);
        }
    });
    vecCls.defineAnnotatedMethods(Vec.class);
  }

Still get same pattern ie class_eval works, regular class monkey patch fails.

Interesting monkey patch instance, as below just works

gem 'minitest' # don't use bundled minitest
require 'java'
require 'minitest/autorun'
require 'minitest/pride'
require_relative '../../monkey/lib/monkey.rb'

Dir.chdir(File.dirname(__FILE__))

module Processing

  class VecTest < Minitest::Test
    def test_rotate
      a = Vec2D.new(3, 5)
      class << a
        def rotate(theta)
          self.x *= Math.cos(theta)
          self.y *= Math.sin(theta)
          self 
        end
      end
      a.rotate(Math::PI / 4)
      b = Vec2D.new(3 * Math.cos(Math::PI / 4), 5 * Math.sin(Math::PI / 4))
      assert_equal(a, b, 'Failed Rotate')
    end 
  end
end

Still failing with:-
jruby 9.1.0.0-SNAPSHOT (2.3.0) 2016-02-27 e657f19 OpenJDK 64-Bit Server VM 25.74-b02 on 1.8.0_74-b02 +jit [linux-amd64]