Autoloading appears to lead to race conditions #3318
Comments
|
Output of This is on Amazon Linux (basically CentOS) running on EC2. We're using Java 1.8, though invoke dynamic was not (explicitly) enabled. |
|
Same happens with rails master (rails 5) on the jruby 9.0.1.0 ( You can reproduce this easily:
It will stuck in the infinite loop somewhere around this method: I've wrapped this method with lock, like this class_attribute :inheritance_mutex
self.inheritance_mutex = Mutex.new
def self.inherited(base) # :nodoc:
inheritance_mutex.synchronize do
base.middleware_stack = middleware_stack.dup
super
end
endAnd that fixed the issue. |
|
Rails 5 edge is able to boot and run on JRuby now, so if the Rails 5 case we indicative this appears to be working fine in 9.1. If someone can reproduce this again we'd be all over investigating it again. |
It appears that JRuby 9.0.1.0 still has problems autoloading constants under certain conditions. We noticed this issue quite a bit on JRuby 1.7.something, to a certain extend it still seems to occur on 9.0.1.0 (though less often).
The problem is basically as following: we run numerous Ruby applications on JRuby for sentiment analysis (various applications bind to Java code, hence JRuby). All these applications use the AWS SDK (a recurring source of headaches at this point). The AWS SDK in turn autoloads pretty much everything, it also dynamically defines a bunch of classes based on AWS API responses if I remember correctly (though this seems unrelated to the problem at hand). In certain instances the AWS code will throw a NameError when it can't find a certain constant. For example (stacktrace is from Rollbar, hence the somewhat odd formatting):
The file in question: https://github.com/aws/aws-sdk-ruby/blob/9eeeb192565e4ed97a427e9e95d99b922274ea42/aws-sdk-core/lib/aws-sdk-core/xml/parser/stack.rb#L48
Autoload is defined here: https://github.com/aws/aws-sdk-ruby/blob/5929ddfb7266961f5acc8fb189685bc1178ba19f/aws-sdk-core/lib/aws-sdk-core/xml/parser.rb#L7-L8
The interesting thing here is that it seems the AWS code is trying to load the constant relative to the wrong root constant. That is, the Frame class is loaded from
Aws::Xml::Parser::Stackwhile it's actually defined inAws::Xml::Parser. Prior to updating this code to version 2 of the AWS SDK I had similar issues where the code was trying to load the classAWS::SQS::Queuebut would end up using the top-levelQueueclass, resulting in an ArgumentError when the class was initialized (asAWS::SQS::Queue#initializehas an arity of 2).Sadly Rollbar didn't actually track what application this error originated from (don't get me started). However, the exception occurs during bootup when it tries to connect to the AWS queue. Since all applications use the same bootup phase this should be possible to reproduce using any of them.
Thankfully the code is (for a change) open source. As such you should be able to reproduce this as following:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_REGIONsetup.rbas noted belowgem install opener-language-identifier, you'll need to have libarchive installed if I remember correctlylanguage-identifier-daemon, terminate it and repeat this process until the error occurs (this is basically where you'll need a lot of luck)setup.rb:
Alternatively, if you want to also test the runtime for any autoloading problems, use the following setup script instead:
Then, instead of terminating the
language-identifier-daemoncommand right away just let it run. If you're on a laptop you might want to take it off your lap, because it's probably going to get very hot.The text was updated successfully, but these errors were encountered: