ConcurrentModification error during Tomcat startup #5115
Milestone
Comments
|
Are you able to trigger this consistently? If possible, you can set JRuby into a debug mode that will show the actual cause of the LoadError better by setting property Very little of JRuby internals would trigger this exception, since most of the time we're using Ruby collection types that raise Ruby errors rather than Java errors like ConcurrentModificationException. With debug mode on we should be able to see what's happening. It would also be helpful if you can make a sample project that we can build and run to reproduce this. A fix can be found within hours if not minutes when we can reproduce the issue locally. |
|
Did I do it wrong? I see |
|
Oh yes, you did! I only looked at the servlet context...did not see the full system properties later. Unfortunately it appears that the error went to a different log. I believe we always dump these Java exceptions during load to whatever the error stream of the JRuby runtime is. Offhand I do not recall where that goes with jruby-rack. Assuming it did print, you should be able to grep other logs for |
|
Found the following in |
|
I don't believe I asked...is this a recent regression? In other words, did it work in 9.1.15 or an earlier version? The exception you found is exactly what I was looking for. Something about the way we are invalidating a particular class hierarchy is causing a concurrency error, either because it is not sufficiently protecting itself from other threads, or because it is improperly manipulating some collection (like deleting while iterating, or similar). So it seems under some circumstance, requiring the I am wondering about the threading characteristics of jruby-rack these days. It is possible that a request is being handled by a runtime at the same time it is loading the Obviously the Is it possible for you to create a dummy application that mimics yours and fails the same way? It would help a great deal if I could reproduce this locally. |
|
The Does |
|
No, Are you able to boot the Rails app on its own, outside of Tomcat? At this point the best I can do is to eyeball the related code and see if I can divine a path through it that might lead to this exception. The big missing bits of information are:
Perhaps you can try to configure your jruby-rack to only have use a separate instance for each request, and see if that solves it? I don't recommend that pattern for deploying, since we and Rails should be safe for concurrent requests, but it would help narrow down the problem. |
|
I may see how this error could be triggered. The concurrent modification is happening against a set of subclasses. That set is itself synchronized (on a per-call basis) but iteration would not be covered by that lock. A thread iterating this list while a subclass is simultaneously removed could trigger the error. |
|
I just discovered |
|
@JasonLunn Ok, that sounds like a good lead. I have also managed to easily reproduce this same exception by following my hunch above: class A; def foo; end; end
Thread.new { loop { Class.new(A) } }
loop { class A; def foo; end; end }This script almost immediately produces the same exception you reported: I am exploring possible fixes. This is a touchy area, so I can't just go throwing locks around. |
headius
added a commit
that referenced
this issue
Apr 12, 2018
The naively-synchronized WeakHashSet does not guarantee safe iteration in the presence of modification, which triggers a ConcurrentModificationException as seen in #5115. By using a ConcurrentWeakHashMap (designed by Doug Lea et al, see http://www.java2s.com/Code/Java/Collections-Data-Structure/Ahashtablewithemweakkeysemfullconcurrencyofretrievalsandadjustableexpectedconcurrencyforupdates.htm) we can iterate and modify concurrently without errors. However this implementation is based the design of ConcurrentHashMap, which means the in-memory structure will likely be considerably larger than a simple WeakHashMap. I believe it is also based off the pre-Java 8 ConcurrentHashMap, which means any load factor above one thread greatly increases in-memory size, and a load factor of one limits the concurrent use of the collection.
|
I've made a PR using a public domain "ConcurrentWeakHashMap" I found that was designed by the java.util.concurrent folks. It allows concurrent traversal and modification, but there are some caveats. |
|
Oh, I also tried another approach on JRuby master: using Java 8 streams to iterate through the subclasses set rather than external iteration. It did not appear to fix the bug. |
|
Ok, you should be able to see this in 9.1 nightlies, though it may be tricky to use them with Warbler until we release 9.1.17. I'm pretty confident this will fix your issue. |
|
When is the next JRubyConf? I owe you two beers this release cycle... |
|
Is Cloudbees' Jenkins down for anyone else? I wanted to try the nightly to see if it would work for me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment
jruby 9.1.16.0 (2.3.3) 2018-02-21 8f3f95a Java HotSpot(TM) 64-Bit Server VM 25.162-b12 on 1.8.0_162-b12 +jit [darwin-x86_64]JAVA_OPTS=-Xmx2G -Djruby.debug.fullTrace=true -Djruby.cli.debug=true -Djruby.backtrace.style=rawExpected Behavior
Actual Behavior
The text was updated successfully, but these errors were encountered: