Error when matching regex in multiple threads #3670

marshalium · 2016-02-15T21:28:06Z

1.7.24 and 9.0.5.0 both have a pretty serious regression when matching against utf8 strings in multiple threads at the same time. This bug does not appear to be present in 1.7.23 or 9.0.4.0.

If multiple threads are running code like this, with completely unshared string and regex objects:

str = "foobar"
str.force_encoding("UTF-8")
str.gsub(/foo/i, '')

eventually one of the threads will throw an error like this:

Exception in thread "Ruby-0-Thread-6: ./recreate_utf8_bug.rb:3" java.lang.ArrayIndexOutOfBoundsException: 6
  at org.jcodings.specific.BaseUTF8Encoding.mbcCaseFold(BaseUTF8Encoding.java:167)
  at org.jcodings.specific.UTF8Encoding.mbcCaseFold(UTF8Encoding.java:24)
  at org.joni.SearchAlgorithm$SLOW_IC.lowerCaseMatch(SearchAlgorithm.java:238)
  at org.joni.SearchAlgorithm$SLOW_IC.search(SearchAlgorithm.java:206)
  at org.joni.Matcher.forwardSearchRange(Matcher.java:140)
  at org.joni.Matcher.searchInterruptible(Matcher.java:451)
  at org.jruby.RubyRegexp$SearchMatchTask.run(RubyRegexp.java:273)
  at org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066)
  at org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:235)
  at org.jruby.RubyString.gsubCommon19(RubyString.java:3123)
  at org.jruby.RubyString.gsubCommon19(RubyString.java:3106)
  at org.jruby.RubyString.gsub19(RubyString.java:3101)
  at org.jruby.RubyString.gsub19(RubyString.java:3069)
  at org.jruby.RubyString$INVOKER$i$gsub19.call(RubyString$INVOKER$i$gsub19.gen)
  at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrTwoOrNBlock.call(JavaMethod.java:367)
  at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
  at $_dot_.recreate_utf8_bug.block_2$RUBY$__file__(./recreate_utf8_bug.rb:13)
  at $_dot_$recreate_utf8_bug$block_2$RUBY$__file__.call($_dot_$recreate_utf8_bug$block_2$RUBY$__file__)
  at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:117)
  at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:92)
  at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
  at org.jruby.RubyFixnum.times(RubyFixnum.java:275)
  at org.jruby.RubyFixnum$INVOKER$i$0$0$times.call(RubyFixnum$INVOKER$i$0$0$times.gen)
  at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
  at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
  at $_dot_.recreate_utf8_bug.chained_0_rescue_1$RUBY$SYNTHETIC__file__(./recreate_utf8_bug.rb:10)
  at $_dot_.recreate_utf8_bug.block_1$RUBY$__file__(./recreate_utf8_bug.rb:9)
  at $_dot_$recreate_utf8_bug$block_1$RUBY$__file__.call($_dot_$recreate_utf8_bug$block_1$RUBY$__file__)
  at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:159)
  at org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:87)
  at org.jruby.runtime.Block.call(Block.java:101)
  at org.jruby.RubyProc.call(RubyProc.java:300)
  at org.jruby.RubyProc.call(RubyProc.java:230)
  at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
  at java.lang.Thread.run(Thread.java:745)

The error doesn't occur if I remove the call to #force_encoding or if I remove the "i" flag from the regex.

See this gist for a full reproducible test case and examples of running it on different JRuby versions:
https://gist.github.com/marshalium/3e62c2affbd2ce95757f

The text was updated successfully, but these errors were encountered:

rsim · 2016-02-22T09:09:59Z

We are experiencing this issue as well and it resulted in a large number of corrupted data in YAML serialized ActiveRecord columns.

rsim · 2016-02-22T11:23:40Z

Based on the original example, I created a simple test case which always fails with ArrayIndexOutOfBoundsException is some threads:

10.times.map{Thread.new{100000.times{"foobar".force_encoding("UTF-8") =~ /foo/i}}}.map(&:value)

I would recommend that http://jruby.org/2016/01/20/jruby-1-7-24 is updated with this critical bug information and with recommendation not to use this version in multi-threaded production applications. This bug most probably will not be caught by application test suites and might cause critical problems in production.

…r instance

lopex · 2016-02-22T19:49:31Z

Temporary fix is in jruby/joni@f23bffa by making SLOW_IC stateless and moving the cached buffer into matcher instance. In the future we need to get rid of those stateful caches and IntHolder by making them local.
@headius @enebo No idea why it might behave as a regression, this was always broken.

lopex · 2016-02-23T18:10:53Z

Commit that revealed that bug in EXACT_IC: jruby/joni@eda1681

headius · 2016-02-23T19:42:04Z

@lopex Perhaps the caching of Regex was not as good and something got fixed around these releases? That would trigger more sharing of this field, wouldn't it?

lopex · 2016-02-23T20:05:25Z

@headius jruby/joni@eda1681 changed the way heuristics applied fast skipping / fast failing by favouring EXACT_IC for this very case. EXACT_IC had a buffer as an instance field and since it itself is scoped by Regexp instance, the buffer essentially turned out to be Regexp global (in case of a literal regexp). The fix just moved that buffer into Matcher instance for potential reuse when jumping in/out of bytecode interpreter. So caching of regexps never had any effect on this.

headius · 2016-02-24T17:12:26Z

I updated ruby-2.3 to use new joni, but we'll need a release for 9.1.

headius · 2016-02-24T17:12:50Z

Oh, and we'll need to apply it to 1.7 of course.

ncuesta · 2016-03-02T17:13:38Z

Hi all, we are experiencing this same issue. Which is the recommended workaround until new versions including a fix for this are released? Should we downgrade to 9.0.4.0 (or 1.7.23 for the 1.7 branch)?

Thanks

enebo · 2016-03-02T17:55:12Z

@ncuesta downgrading I think is the safest bet until we put out new releases. We will hopefully have a new release out in the next week to 2 weeks at most. At this point we are tidying up after the last dev cycle...

ncuesta · 2016-03-02T19:23:01Z

Thanks @enebo!

colinsurprenant · 2016-04-01T19:28:44Z

we also started to notice this in the new logstash 2.3 release which includes 1.7.24, in useragent filter logstash-plugins/logstash-filter-useragent#25 and we are tracking this regex issue in elastic/logstash#4977

enebo · 2016-04-11T16:30:27Z

joni has been updated and released for a few weeks with this addressed. Closing.

MartinKoerner · 2016-04-18T14:33:05Z

Will there be a new 9.0.x release for this fix or do we have to wait until 9.1?

The JDBC MSSQL adapter code is full of //i Regex expressions, which produce 'nice' exceptions like 'no columns for table: ' or 'undefined method `[]' for nil:NilClass'

enebo · 2016-04-18T14:49:50Z

@MartinKoerner There will only be a 9.1 release which should be this week...

enebo · 2016-05-11T18:29:51Z

@DzmitryNikitsin I see this comment sent to me as email but not in github issues. Was this
comment withdrawn?

On Fri, May 6, 2016 at 9:15 PM, DzmitryNikitsin notifications@github.com
wrote:

@enebo https://github.com/enebo i've just updated to 9.1.0.0, using
regexp match with multithreading, still seeing the same issue :(

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3670 (comment)

blog: http://blog.enebo.com twitter: tom_enebo
mail: tom.enebo@gmail.com

DzmitryNikitsin · 2016-05-11T18:39:56Z

Yes, issue was fixed on 9.1.0.0.

Thanks,
Dzmitry

From: Thomas E Enebo notifications@github.com
Reply-To: jruby/jruby reply@reply.github.com
Date: Wednesday, May 11, 2016 at 2:30 PM
To: jruby/jruby jruby@noreply.github.com
Cc: Dzmitry Nikitsin dzmitry.nikitsin@gmail.com, Comment comment@noreply.github.com
Subject: Re: [jruby/jruby] Error when matching regex in multiple threads (#3670)

I see this comment sent to me as email but not in github issues. Was this
comment withdrawn?

On Fri, May 6, 2016 at 9:15 PM, DzmitryNikitsin notifications@github.com
wrote:

@enebo https://github.com/enebo i've just updated to 9.1.0.0, using
regexp match with multithreading, still seeing the same issue :(

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#3670 (comment)

blog: http://blog.enebo.com twitter: tom_enebo
mail: tom.enebo@gmail.com

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

enebo added this to the JRuby 1.7.25 milestone Feb 15, 2016

enebo added the core label Feb 15, 2016

kares added the regression label Feb 16, 2016

lopex added a commit to jruby/joni that referenced this issue Feb 22, 2016

temporary fix for jruby/jruby#3670 by moving buffer caching to matche…

f23bffa

…r instance

headius mentioned this issue Mar 4, 2016

JRuby 1.7.24 - REXML bad encoding issue #3632

Closed

sonots mentioned this issue Mar 18, 2016

[embulk 0.8.8, 0.8.9] java.lang.ArrayIndexOutOfBoundsException at org.jcodings.specific.BaseUTF8Encoding.mbcCaseFold(BaseUTF8Encoding.java:167) embulk/embulk-output-bigquery#38

Closed

colinsurprenant mentioned this issue Apr 1, 2016

JRuby 1.7.24 regex thread safety regression elastic/logstash#4977

Closed

lopex mentioned this issue Apr 2, 2016

9.0.5.0 ArrayIndexOutOfBoundsException in BaseUTF8Encoding.java:167 #3776

Closed

colinsurprenant mentioned this issue Apr 4, 2016

revert to jruby 1.7.23 for regex regression elastic/logstash#4995

Merged

enebo closed this as completed Apr 11, 2016

kares mentioned this issue Apr 20, 2016

JRuby bundler regression in 9.0.5.0 #3635

Closed

sl-victorceron mentioned this issue Jan 30, 2017

EmrEtlRunner: race condition overwriting Clojure Collector files during staging step snowplow/snowplow#3085

Closed

hiroyuki-sato mentioned this issue Apr 14, 2017

Update JRuby 9.1.8.0 embulk/embulk#600

Closed

nightpool mentioned this issue Aug 1, 2019

Add option to exclude suspended domains/subdomains from tootctl domains crawl mastodon/mastodon#11454

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Sponsors

Error when matching regex in multiple threads #3670

Error when matching regex in multiple threads #3670

marshalium commented Feb 15, 2016

rsim commented Feb 22, 2016

rsim commented Feb 22, 2016

lopex commented Feb 22, 2016

lopex commented Feb 23, 2016

headius commented Feb 23, 2016

lopex commented Feb 23, 2016

headius commented Feb 24, 2016

headius commented Feb 24, 2016

ncuesta commented Mar 2, 2016

enebo commented Mar 2, 2016

ncuesta commented Mar 2, 2016

colinsurprenant commented Apr 1, 2016

enebo commented Apr 11, 2016

MartinKoerner commented Apr 18, 2016

enebo commented Apr 18, 2016

enebo commented May 11, 2016 •

edited

Loading

DzmitryNikitsin commented May 11, 2016

Error when matching regex in multiple threads #3670

Error when matching regex in multiple threads #3670

Comments

marshalium commented Feb 15, 2016

rsim commented Feb 22, 2016

rsim commented Feb 22, 2016

lopex commented Feb 22, 2016

lopex commented Feb 23, 2016

headius commented Feb 23, 2016

lopex commented Feb 23, 2016

headius commented Feb 24, 2016

headius commented Feb 24, 2016

ncuesta commented Mar 2, 2016

enebo commented Mar 2, 2016

ncuesta commented Mar 2, 2016

colinsurprenant commented Apr 1, 2016

enebo commented Apr 11, 2016

MartinKoerner commented Apr 18, 2016

enebo commented Apr 18, 2016

enebo commented May 11, 2016 • edited Loading

DzmitryNikitsin commented May 11, 2016

enebo commented May 11, 2016 •

edited

Loading