Works ok on master, but that's not too surprising (reimplemented encodings since 1.7).

I would suspect the problem is in our Regexp encoding negotiation (prepareRegexp and such); it's not allowing this combination when it should.

Here's the stack trace:

Encoding::CompatibilityError: incompatible character encodings: US-ASCII and UTF-16LE
    from org/jruby/RubyString.java:1228:in `%'
    from org/jruby/RubyKernel.java:986:in `sprintf'
    from org/jruby/RubyKernel.java:615:in `printf'
    from org/jruby/RubyKernel.java:1511:in `loop'
    from org/jruby/RubyKernel.java:1274:in `catch'
    from org/jruby/RubyKernel.java:1274:in `catch'
    from /Users/azolotko/.rbenv/versions/jruby-1.7.20/bin/jirb:13:in `(root)'

As you can see, the error is raised when IRB tries to printf the result of the evaluation, i.e. => /abc/.

The actual difference between JRuby and MRI seems to be in the result of the Regexp#inspect:
regex16.inspect in JRuby 1.7.20 returns "\u612F\u6200\u6300\u2F00", when MRI does "/a\x00b\x00c\x00/"

JRuby 1.7.20:

>> regex16.inspect.encoding
=> #<Encoding:UTF-16LE>

MRI 2.2.2p95:

irb(main):004:0> regex16.inspect.encoding
=> #<Encoding:US-ASCII>

MRI forces US-ASCII if the Regexp encoding is not compatible with ASCII: https://github.com/ruby/ruby/blob/ruby_2_2/re.c#L434
https://github.com/ruby/ruby/blob/ruby_2_2/re.c#L411

But JRuby uses the original encoding:
https://github.com/jruby/jruby/blob/jruby-1_7/core/src/main/java/org/jruby/RubyRegexp.java#L1843
https://github.com/jruby/jruby/blob/jruby-1_7/core/src/main/java/org/jruby/RubyRegexp.java#L1314

@azolotko you did such a good job find this is there any chance you can submit a pull request too?

@enebo appendRegexpString19 looks scary. Let's see if I can do this.

Sorry guys, I didn't manage to tackle the issue and can't spend more time on this. Maybe someone else will have a better luck.

Ok, I am not going to fix this before 1.7.25 goes out but I learned some stuff so I will document something at least:

abc8 = "abc"
p abc8.encoding  # <Encoding:UTF-8>
regex8 = Regexp.new(abc8)  # succeeds

p regex8.inspect
p regex8.inspect.encoding


abc16 = "abc".encode("UTF-16LE")
p abc16.encoding  # <Encoding:UTF-16LE>
regex16 = Regexp.new(abc16)

p regex16.inspect
p regex16.inspect.encoding

Both 1.7 and 9k are both wrong. 9k does not throw the original reported exception but it is still not properly converting the encoding to the same one as MRI. I did just move over the 9k logic from regexpDescription to 1.7 as a test and the exception goes away but the result is still wrong; so I do not think that is appropriate to unbreak it less.

Results:

(mri 1.9)

#<Encoding:US-ASCII>
"/abc/"
#<Encoding:US-ASCII>
#<Encoding:UTF-16LE>
"/a\x00b\x00c\x00/"
#<Encoding:US-ASCII>

mri2.3

#<Encoding:UTF-8>
"/abc/"
#<Encoding:US-ASCII>
#<Encoding:UTF-16LE>
"/a\x00b\x00c\x00/"
#<Encoding:US-ASCII>

Only difference in MRI is that default encoding changed so we see UTF-8 in first print out.

JRuby 1.7.24 has original reported problem Encoding::ComatibilityError.

9.0.5.0

#<Encoding:UTF-8>
"/abc/"
#<Encoding:UTF-8>
#<Encoding:UTF-16LE>
"\u612F\u6200\u6300\u2F00"
#<Encoding:UTF-16LE>

So difference is as @azolotko said inspect will convert the inspect of a regexp to US-ASCII and then print out mbc's as /x codes.

This works on 1.7 HEAD, so I'll call it fixed in 1.7.26.