Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jit failure: rack #4255

Closed
fedenusy opened this issue Oct 28, 2016 · 4 comments
Closed

jit failure: rack #4255

fedenusy opened this issue Oct 28, 2016 · 4 comments
Labels
jit JRuby 9000 needs tests regression
Milestone
JRuby 9.1.6.0

Comments

@fedenusy
Copy link

Environment

  • mac os x el capitan v10.11.6
  • java -version:
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • jruby 9.1.5.0 (via rvm)
  • rails 4.2.0 app packaged with warbler
    • use warbler master branch, which includes a bundler fix
    • config.features = %w(gemjar runnable executable)
    • config.override_gem_home = true
  • run with:
java -server -Djruby.compile.mode=JIT -Djruby.jit.logging=true -Djruby.jit.logging.verbose=true -Djruby.jit.threshold=0 -jar app.war

Expected Behavior

methods should get jitted on startup

Actual Behavior

many methods get jitted, then:

2016-10-27T18:13:30.413-07:00 [main] INFO JITCompiler : Could not compile; passes run: [org.jruby.ir.passes.LocalOptimizationPass@584ed071, org.jruby.ir.passes.DeadCodeElimination@805d30ee, org.jruby.ir.passes.OptimizeDynScopesPass@f61f6f63, org.jruby.ir.passes.OptimizeDelegationPass@6c7a3ce9, org.jruby.ir.passes.AddCallProtocolInstructions@87f2d916, org.jruby.ir.passes.AddMissingInitsPass@a7c39fef]: <block> null_CLOSURE_1.null_CLOSURE_1 at null:0 because of: "null"
java.lang.NullPointerException
    at org.jruby.util.JavaNameMangler.mangleMethodNameInternal(JavaNameMangler.java:233)
    at org.jruby.util.JavaNameMangler.encodeScopeForBacktrace(JavaNameMangler.java:297)
    at org.jruby.ir.targets.JVMVisitor.emitBlockJIT(JVMVisitor.java:294)
    at org.jruby.ir.targets.JVMVisitor.codegenScope(JVMVisitor.java:119)
    at org.jruby.ir.targets.JVMVisitor.compileToBytecode(JVMVisitor.java:89)
    at org.jruby.compiler.JITCompiler$BlockJITClassGenerator.compile(JITCompiler.java:554)
    at org.jruby.compiler.JITCompiler$BlockJITTask.run(JITCompiler.java:358)
    at org.jruby.compiler.JITCompiler.buildThresholdReached(JITCompiler.java:195)
    at org.jruby.runtime.MixedModeIRBlockBody.promoteToFullBuild(MixedModeIRBlockBody.java:183)
    at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:126)
    at org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:119)
    at org.jruby.runtime.Block.yieldNonArray(Block.java:171)
    at org.jruby.RubyBasicObject.yieldUnder(RubyBasicObject.java:1764)
    at org.jruby.RubyBasicObject.specificEval(RubyBasicObject.java:1787)
    at org.jruby.RubyBasicObject.instance_eval19(RubyBasicObject.java:1667)
    at org.jruby.RubyBasicObject$INVOKER$i$instance_eval19.call(RubyBasicObject$INVOKER$i$instance_eval19.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:328)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:141)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at uri_3a_classloader_3a_.gems.rack_minus_1_dot_6_dot_4.lib.rack.builder.invokeOther5:instance_eval(uri:classloader:/gems/rack-1.6.4/lib/rack/builder.rb:55)
    at uri_3a_classloader_3a_.gems.rack_minus_1_dot_6_dot_4.lib.rack.builder.RUBY$method$initialize$0(uri:classloader:/gems/rack-1.6.4/lib/rack/builder.rb:55)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.RubyClass.newInstance(RubyClass.java:1022)
    at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:356)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:869)
    at org.jruby.Ruby.runInterpreter(Ruby.java:874)
    at org.jruby.Ruby.executeScript(Ruby.java:508)
    at org.jruby.rack.DefaultRackApplicationFactory.createRackServletWrapper(DefaultRackApplicationFactory.java:239)
    at org.jruby.rack.DefaultRackApplicationFactory.createRackServletWrapper(DefaultRackApplicationFactory.java:228)
    at org.jruby.rack.rails.RailsRackApplicationFactory.createApplicationObject(RailsRackApplicationFactory.java:23)
    at org.jruby.rack.DefaultRackApplicationFactory$1.create(DefaultRackApplicationFactory.java:101)
    at org.jruby.rack.DefaultRackApplicationFactory$RackApplicationImpl.init(DefaultRackApplicationFactory.java:445)
    at org.jruby.rack.DefaultRackApplicationFactory.getApplication(DefaultRackApplicationFactory.java:114)
    at org.jruby.rack.SharedRackApplicationFactory.doInit(SharedRackApplicationFactory.java:34)
    at org.jruby.rack.RackApplicationFactoryDecorator.init(RackApplicationFactoryDecorator.java:100)
    at org.jruby.rack.RackServletContextListener.contextInitialized(RackServletContextListener.java:50)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.runner.Runner.run(Runner.java:509)
    at org.eclipse.jetty.runner.Runner.main(Runner.java:557)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at WarMain.launchWebServer(WarMain.java:187)
    at WarMain.start(WarMain.java:345)
    at JarMain.doStart(JarMain.java:233)
    at WarMain.main(WarMain.java:367)

the error seems to halt startup jitting altogether
@headius
Copy link
Member

headius commented Oct 30, 2016

Neat. Guess we need better handling for errors during JIT prep, as well as more robust name-mangling code.

@headius headius added this to the JRuby 9.1.6.0 milestone Oct 30, 2016
headius added a commit that referenced this issue Oct 30, 2016
@headius
Wrap all runtime JIT in try/catch to help cases like #4255.
35f4547
@headius
Copy link
Member

headius commented Oct 30, 2016

I have pushed better exception-handling in 35f4547, but I have not been able to reproduce locally. Can you try to run this again with a JRuby master build, adding some logging into rack/lib/builder.rb:55 to see where that block is coming from?

@fedenusy
Copy link
Author

fedenusy commented Nov 2, 2016

with master build i'm seeing a new compile error:

2016-11-02T15:17:08.571-07:00 [main] INFO JITCompiler : Could not compile; passes run: [org.jruby.ir.passes.LocalOptimizationPass@584ed071, org.jruby.ir.passes.DeadCodeElimination@805d30ee, org.jruby.ir.passes.OptimizeDynScopesPass@f61f6f63, org.jruby.ir.passes.OptimizeDelegationPass@6c7a3ce9, org.jruby.ir.passes.AddCallProtocolInstructions@87f2d916, org.jruby.ir.passes.AddMissingInitsPass@a7c39fef]: <block> uri:classloader:/gems/phony-2.15.26/lib/phony/countries.rb_CLOSURE_1.uri:classloader:/gems/phony-2.15.26/lib/phony/countries.rb_CLOSURE_1 at uri:classloader:/gems/phony-2.15.26/lib/phony/countries.rb:43 because of: "Could not compile org.jruby.runtime.MixedModeIRBlockBody@2cfc81f8; instruction count 2315 exceeds threshold of 2000"
org.jruby.compiler.NotCompilableException: Could not compile org.jruby.runtime.MixedModeIRBlockBody@2cfc81f8; instruction count 2315 exceeds threshold of 2000
    at org.jruby.compiler.JITCompiler$BlockJITClassGenerator.compile(JITCompiler.java:553)
    at org.jruby.compiler.JITCompiler$BlockJITTask.run(JITCompiler.java:362)
    at org.jruby.compiler.JITCompiler.buildThresholdReached(JITCompiler.java:196)
    at org.jruby.runtime.MixedModeIRBlockBody.promoteToFullBuild(MixedModeIRBlockBody.java:183)
    at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:126)
    at org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:119)
    at org.jruby.runtime.Block.yieldNonArray(Block.java:171)
    at org.jruby.RubyBasicObject.yieldUnder(RubyBasicObject.java:1782)
    at org.jruby.RubyBasicObject.specificEval(RubyBasicObject.java:1805)
    at org.jruby.RubyBasicObject.instance_eval19(RubyBasicObject.java:1685)
    at org.jruby.RubyBasicObject$INVOKER$i$instance_eval19.call(RubyBasicObject$INVOKER$i$instance_eval19.gen)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at uri_3a_classloader_3a_.gems.phony_minus_2_dot_15_dot_26.lib.phony.dsl.invokeOther3:instance_eval(uri:classloader:/gems/phony-2.15.26/lib/phony/dsl.rb:17)
    at uri_3a_classloader_3a_.gems.phony_minus_2_dot_15_dot_26.lib.phony.dsl.RUBY$method$define$0(uri:classloader:/gems/phony-2.15.26/lib/phony/dsl.rb:17)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:356)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:862)
    at org.jruby.Ruby.loadFile(Ruby.java:2960)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:885)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:525)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:396)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:966)
    at org.jruby.RubyKernel.require19(RubyKernel.java:959)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneBlock.call(JavaMethod.java:582)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:101)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1008)
    at org.jruby.ir.runtime.IRRuntimeHelpers.zSuper(IRRuntimeHelpers.java:1021)
    at uri_3a_classloader_3a_.gems.activesupport_minus_4_dot_2_dot_0.lib.active_support.dependencies.invokeSuper7:-unknown-super-target-(uri:classloader:/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274)
    at uri_3a_classloader_3a_.gems.activesupport_minus_4_dot_2_dot_0.lib.active_support.dependencies.RUBY$block$require$2(uri:classloader:/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:73)
    at org.jruby.runtime.Block.yieldSpecific(Block.java:136)
    at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:454)
    at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:114)
    at uri_3a_classloader_3a_.gems.activesupport_minus_4_dot_2_dot_0.lib.active_support.dependencies.RUBY$method$load_dependency$0(uri:classloader:/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:240)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
    at uri_3a_classloader_3a_.gems.activesupport_minus_4_dot_2_dot_0.lib.active_support.dependencies.invokeOther9:load_dependency(uri:classloader:/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274)
    at uri_3a_classloader_3a_.gems.activesupport_minus_4_dot_2_dot_0.lib.active_support.dependencies.RUBY$method$require$0(uri:classloader:/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:862)
    at org.jruby.Ruby.loadFile(Ruby.java:2960)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:885)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:525)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:396)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:966)
    at org.jruby.RubyKernel.require19(RubyKernel.java:959)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:383)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.runtime.invokeOther2:require(uri:classloader:/gems/bundler-1.13.6/lib/bundler/runtime.rb:91)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.runtime.RUBY$block$require$2(uri:classloader:/gems/bundler-1.13.6/lib/bundler/runtime.rb:91)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
    at org.jruby.runtime.Block.yield(Block.java:167)
    at org.jruby.specialized.RubyArrayOneObject.each(RubyArrayOneObject.java:128)
    at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.runtime.invokeOther22:each(uri:classloader:/gems/bundler-1.13.6/lib/bundler/runtime.rb:86)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.runtime.RUBY$block$require$1(uri:classloader:/gems/bundler-1.13.6/lib/bundler/runtime.rb:86)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
    at org.jruby.runtime.Block.yield(Block.java:167)
    at org.jruby.RubyArray.each(RubyArray.java:1734)
    at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:328)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:141)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.runtime.invokeOther48:each(uri:classloader:/gems/bundler-1.13.6/lib/bundler/runtime.rb:75)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.runtime.RUBY$method$require$0(uri:classloader:/gems/bundler-1.13.6/lib/bundler/runtime.rb:75)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:173)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:308)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:69)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.invokeOther1:require(uri:classloader:/gems/bundler-1.13.6/lib/bundler.rb:106)
    at uri_3a_classloader_3a_.gems.bundler_minus_1_dot_13_dot_6.lib.bundler.RUBY$method$require$0(uri:classloader:/gems/bundler-1.13.6/lib/bundler.rb:106)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:104)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:862)
    at org.jruby.Ruby.loadFile(Ruby.java:2960)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:885)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:525)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:396)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:966)
    at org.jruby.RubyKernel.require19(RubyKernel.java:959)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:383)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.invokeOther82:gem_original_require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55)
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.RUBY$method$require$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:862)
    at org.jruby.Ruby.loadFile(Ruby.java:2960)
    at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
    at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
    at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:885)
    at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:525)
    at org.jruby.runtime.load.LoadService.require(LoadService.java:396)
    at org.jruby.RubyKernel.requireCommon(RubyKernel.java:966)
    at org.jruby.RubyKernel.require19(RubyKernel.java:959)
    at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:383)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.invokeOther82:gem_original_require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55)
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.RUBY$method$require$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:55)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at uri_3a_classloader_3a_.jruby.rack.rails.environment3.invokeOther12:require(uri:classloader:/jruby/rack/rails/environment3.rb:23)
    at uri_3a_classloader_3a_.jruby.rack.rails.environment3.RUBY$method$load_environment$0(uri:classloader:/jruby/rack/rails/environment3.rb:23)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri_3a_classloader_3a_.jruby.rack.rails_booter.invokeOther3:load_environment(uri:classloader:/jruby/rack/rails_booter.rb:83)
    at uri_3a_classloader_3a_.jruby.rack.rails_booter.RUBY$method$load_environment$0(uri:classloader:/jruby/rack/rails_booter.rb:83)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:340)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.evalScriptlet(Ruby.java:480)
    at org.jruby.Ruby.evalScriptlet(Ruby.java:456)
    at org.jruby.rack.rails.RailsRackApplicationFactory.createApplicationObject(RailsRackApplicationFactory.java:22)
    at org.jruby.rack.DefaultRackApplicationFactory$1.create(DefaultRackApplicationFactory.java:101)
    at org.jruby.rack.DefaultRackApplicationFactory$RackApplicationImpl.init(DefaultRackApplicationFactory.java:445)
    at org.jruby.rack.DefaultRackApplicationFactory.getApplication(DefaultRackApplicationFactory.java:114)
    at org.jruby.rack.SharedRackApplicationFactory.doInit(SharedRackApplicationFactory.java:34)
    at org.jruby.rack.RackApplicationFactoryDecorator.init(RackApplicationFactoryDecorator.java:100)
    at org.jruby.rack.RackServletContextListener.contextInitialized(RackServletContextListener.java:50)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.runner.Runner.run(Runner.java:509)
    at org.eclipse.jetty.runner.Runner.main(Runner.java:557)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at WarMain.launchWebServer(WarMain.java:187)
    at WarMain.start(WarMain.java:345)
    at JarMain.doStart(JarMain.java:233)
    at WarMain.main(WarMain.java:367)

and this one:

2016-11-02T15:33:56.940-07:00 [pool-1-thread-3] INFO JITCompiler : Could not compile; passes run: [org.jruby.ir.passes.LocalOptimizationPass@584ed071, org.jruby.ir.passes.DeadCodeElimination@805d30ee, org.jruby.ir.passes.OptimizeDynScopesPass@f61f6f63, org.jruby.ir.passes.OptimizeDelegationPass@6c7a3ce9, org.jruby.ir.passes.AddCallProtocolInstructions@87f2d916, org.jruby.ir.passes.AddMissingInitsPass@a7c39fef]: HashWithIndifferentAccess ActiveSupport::HashWithIndifferentAccess.to_hash at uri:classloader:/gems/activesupport-4.2.0/lib/active_support/hash_with_indifferent_access.rb:248 because of: "null"
java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
    at java.util.ArrayList$Itr.next(ArrayList.java:851)
    at org.jruby.ir.IRScope.setUpUseDefLocalVarMaps(IRScope.java:942)
    at org.jruby.ir.dataflow.analyses.DefinedVariableNode.identifyUndefinedVarsInClosure(DefinedVariableNode.java:91)
    at org.jruby.ir.dataflow.analyses.DefinedVariableNode.identifyInits(DefinedVariableNode.java:145)
    at org.jruby.ir.dataflow.analyses.DefinedVariablesProblem.findUndefinedVars(DefinedVariablesProblem.java:45)
    at org.jruby.ir.passes.AddMissingInitsPass.execute(AddMissingInitsPass.java:28)
    at org.jruby.ir.passes.CompilerPass.run(CompilerPass.java:111)
    at org.jruby.ir.passes.CompilerPass.run(CompilerPass.java:125)
    at org.jruby.ir.IRScope.runCompilerPasses(IRScope.java:502)
    at org.jruby.ir.IRScope.prepareForCompilation(IRScope.java:586)
    at org.jruby.ir.IRMethod.prepareForCompilation(IRMethod.java:50)
    at org.jruby.ir.targets.JVMVisitor.emitScope(JVMVisitor.java:133)
    at org.jruby.ir.targets.JVMVisitor.emitWithSignatures(JVMVisitor.java:310)
    at org.jruby.ir.targets.JVMVisitor.emitMethodJIT(JVMVisitor.java:287)
    at org.jruby.ir.targets.JVMVisitor.codegenScope(JVMVisitor.java:118)
    at org.jruby.ir.targets.JVMVisitor.compileToBytecode(JVMVisitor.java:90)
    at org.jruby.compiler.JITCompiler$MethodJITClassGenerator.compile(JITCompiler.java:463)
    at org.jruby.compiler.JITCompiler$MethodJITTask.run(JITCompiler.java:271)
    at org.jruby.compiler.JITCompiler.buildThresholdReached(JITCompiler.java:196)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.tryJit(MixedModeIRMethod.java:289)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:124)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.factory.invokeOther16:to_hash(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients/factory.rb:74)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.factory.RUBY$method$options$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients/factory.rb:74)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.factory.invokeOther23:options(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients/factory.rb:64)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.factory.RUBY$method$create_client$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients/factory.rb:64)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:84)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.invokeOther0:mongo_client(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients.rb:135)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.RUBY$method$collection$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients.rb:135)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at org.jruby.RubyClass.finvoke(RubyClass.java:522)
    at org.jruby.RubyBasicObject.send19(RubyBasicObject.java:1678)
    at org.jruby.RubyKernel.send19(RubyKernel.java:1999)
    at org.jruby.RubyKernel$INVOKER$s$send19.call(RubyKernel$INVOKER$s$send19.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.options.invokeOther4:send(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients/options.rb:283)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.clients.options.RUBY$method$method_missing$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/clients/options.rb:283)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at org.jruby.runtime.Helpers$MethodMissingMethod.call(Helpers.java:355)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:195)
    at org.jruby.runtime.callsite.CachingCallSite.callMethodMissing(CachingCallSite.java:405)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:315)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.contextual.mongo.invokeOther29:collection(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/contextual/mongo.rb:318)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.contextual.mongo.RUBY$method$initialize$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/contextual/mongo.rb:318)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:348)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:173)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)
    at org.jruby.RubyClass.newInstance(RubyClass.java:1001)
    at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.contextual.invokeOther9:new(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/contextual.rb:53)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.contextual.RUBY$method$create_context$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/contextual.rb:53)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.contextual.invokeOther4:create_context(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/contextual.rb:35)
    at uri_3a_classloader_3a_.gems.mongoid_minus_5_dot_1_dot_5.lib.mongoid.contextual.RUBY$method$context$0(uri:classloader:/gems/mongoid-5.1.5/lib/mongoid/contextual.rb:35)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri.invokeOther0:context(uri:65535)
    at uri.RUBY$method$first$0(uri)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:93)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.mongoid.virtual_collection.invokeOther35:first(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/mongoid/virtual_collection.rb:64)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.mongoid.virtual_collection.RUBY$block$sync_virtual_collection?$2(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/mongoid/virtual_collection.rb:64)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:73)
    at org.jruby.runtime.Block.yieldSpecific(Block.java:136)
    at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:454)
    at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:114)
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.monitor.RUBY$method$mon_synchronize$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:214)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:81)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:328)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:141)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.mongoid.virtual_collection.invokeOther55:synchronize(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/mongoid/virtual_collection.rb:61)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.mongoid.virtual_collection.RUBY$method$sync_virtual_collection?$0(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/mongoid/virtual_collection.rb:61)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.mongoid.virtual_collection.invokeOther5:sync_virtual_collection?(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/mongoid/virtual_collection.rb:24)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.mongoid.virtual_collection.RUBY$method$find_in_memory$0(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/mongoid/virtual_collection.rb:24)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:236)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.d_models.d_featured_page.invokeOther6:find_in_memory(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/d_models/d_featured_page.rb:21)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.d_models.d_featured_page.RUBY$block$ensure_locale_coverage$2(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/d_models/d_featured_page.rb:21)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
    at org.jruby.runtime.Block.yield(Block.java:167)
    at org.jruby.RubyArray.each(RubyArray.java:1734)
    at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:328)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:141)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.d_models.d_featured_page.invokeOther11:each(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/d_models/d_featured_page.rb:20)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.d_models.d_featured_page.RUBY$method$ensure_locale_coverage$0(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/d_models/d_featured_page.rb:20)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.workers.in_app_worker.invokeOther2:ensure_locale_coverage(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/workers/in_app_worker.rb:86)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.workers.in_app_worker.RUBY$method$prepare_featured_pages$0(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/workers/in_app_worker.rb:86)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.workers.in_app_worker.invokeOther2:prepare_featured_pages(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/workers/in_app_worker.rb:35)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.workers.in_app_worker.RUBY$block$initialize$3(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/workers/in_app_worker.rb:35)
    at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
    at org.jruby.runtime.Block.call(Block.java:126)
    at org.jruby.RubyProc.call(RubyProc.java:324)
    at org.jruby.RubyProc.call19(RubyProc.java:308)
    at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:173)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:67)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.safe_task_executor.invokeOther1:call(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/safe_task_executor.rb:24)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.safe_task_executor.RUBY$block$execute$1(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/safe_task_executor.rb:24)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
    at org.jruby.runtime.Block.yield(Block.java:167)
    at com.concurrent_ruby.ext.SynchronizationLibrary$JRubyLockableObject.rubySynchronize(SynchronizationLibrary.java:174)
    at com.concurrent_ruby.ext.SynchronizationLibrary$JRubyLockableObject$INVOKER$i$0$0$rubySynchronize.call(SynchronizationLibrary$JRubyLockableObject$INVOKER$i$0$0$rubySynchronize.gen)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.safe_task_executor.invokeOther4:synchronize(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/safe_task_executor.rb:19)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.safe_task_executor.RUBY$method$execute$0(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/safe_task_executor.rb:19)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:104)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:199)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
    at org.jruby.runtime.MethodBlockBody.call(MethodBlockBody.java:60)
    at org.jruby.runtime.Block.call(Block.java:126)
    at org.jruby.RubyProc.call(RubyProc.java:324)
    at org.jruby.RubyProc.call19(RubyProc.java:308)
    at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:356)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
    at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
    at org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:179)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:110)
    at org.jruby.runtime.Block.yield(Block.java:167)
    at com.concurrent_ruby.ext.SynchronizationLibrary$JRubyLockableObject.rubySynchronize(SynchronizationLibrary.java:174)
    at com.concurrent_ruby.ext.SynchronizationLibrary$JRubyLockableObject$INVOKER$i$0$0$rubySynchronize.call(SynchronizationLibrary$JRubyLockableObject$INVOKER$i$0$0$rubySynchronize.gen)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:497)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:356)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:173)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:67)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.ivar.invokeOther6:execute(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/ivar.rb:170)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.ivar.RUBY$method$safe_execute$0(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/ivar.rb:170)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:113)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:198)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:207)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.scheduled_task.invokeOther5:safe_execute(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/scheduled_task.rb:285)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.scheduled_task.RUBY$method$process_task$0(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/scheduled_task.rb:285)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:223)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:97)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.timer_set.invokeOther18:process_task(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/timer_set.rb:99)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.timer_set.RUBY$block$ns_post_task$2(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/timer_set.rb:99)
    at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
    at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
    at org.jruby.runtime.Block.call(Block.java:126)
    at org.jruby.RubyProc.call(RubyProc.java:324)
    at org.jruby.RubyProc.call19(RubyProc.java:308)
    at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:173)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:67)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.java_executor_service.invokeOther2:call(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/java_executor_service.rb:94)
    at uri_3a_classloader_3a_.gems.concurrent_minus_ruby_minus_1_dot_0_dot_3_dot_pre3_minus_java.lib.concurrent.executor.java_executor_service.RUBY$method$run$0(uri:classloader:/gems/concurrent-ruby-1.0.3.pre3-java/lib/concurrent/executor/java_executor_service.rb:94)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at Concurrent$$JavaExecutorService$$Job_249529271.run(Concurrent$$JavaExecutorService$$Job_249529271.gen:13)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

other methods keep getting jitted on startup, even after these errors get raised 👍

i tried overriding Rack::Builder.initialize to add logging (this is the method defined at rack/lib/builder.rb:55). unfortunately, the monkey patch triggers a new error:

2016-11-02 15:17:59.090:WARN:/:main: ERROR: initialization failed
org.jruby.rack.RackInitializationException: java.lang.NullPointerException
    at org.jruby.rack.RackInitializationException.wrap(RackInitializationException.java:31)
    at org.jruby.rack.RackApplicationFactoryDecorator.init(RackApplicationFactoryDecorator.java:104)
    at org.jruby.rack.RackServletContextListener.contextInitialized(RackServletContextListener.java:50)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.runner.Runner.run(Runner.java:509)
    at org.eclipse.jetty.runner.Runner.main(Runner.java:557)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at WarMain.launchWebServer(WarMain.java:187)
    at WarMain.start(WarMain.java:345)
    at JarMain.doStart(JarMain.java:233)
    at WarMain.main(WarMain.java:367)
Caused by: 
java.lang.NullPointerException
    at org.jruby.runtime.backtrace.RubyStackTraceElement.mriStyleString(RubyStackTraceElement.java:77)
    at org.jruby.runtime.ThreadContext.createCallerBacktrace(ThreadContext.java:699)
    at org.jruby.RubyKernel.caller20(RubyKernel.java:1063)
    at org.jruby.RubyKernel$INVOKER$s$0$2$caller20.call(RubyKernel$INVOKER$s$0$2$caller20.gen)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:195)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.rack.builder.invokeOther4:caller(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/rack/builder.rb:7)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.rack.builder.RUBY$method$initialize$0(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/rack/builder.rb:7)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.invokeExact(CompiledIRMethod.java:210)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:88)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.RubyClass.newInstance(RubyClass.java:1022)
    at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
    at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:356)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
    at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
    at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
    at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
    at org.jruby.Ruby.runInterpreter(Ruby.java:867)
    at org.jruby.Ruby.runInterpreter(Ruby.java:871)
    at org.jruby.Ruby.executeScript(Ruby.java:506)
    at org.jruby.rack.DefaultRackApplicationFactory.createRackServletWrapper(DefaultRackApplicationFactory.java:239)
    at org.jruby.rack.DefaultRackApplicationFactory.createRackServletWrapper(DefaultRackApplicationFactory.java:228)
    at org.jruby.rack.rails.RailsRackApplicationFactory.createApplicationObject(RailsRackApplicationFactory.java:23)
    at org.jruby.rack.DefaultRackApplicationFactory$1.create(DefaultRackApplicationFactory.java:101)
    at org.jruby.rack.DefaultRackApplicationFactory$RackApplicationImpl.init(DefaultRackApplicationFactory.java:445)
    at org.jruby.rack.DefaultRackApplicationFactory.getApplication(DefaultRackApplicationFactory.java:114)
    at org.jruby.rack.SharedRackApplicationFactory.doInit(SharedRackApplicationFactory.java:34)
    at org.jruby.rack.RackApplicationFactoryDecorator.init(RackApplicationFactoryDecorator.java:100)
    at org.jruby.rack.RackServletContextListener.contextInitialized(RackServletContextListener.java:50)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.runner.Runner.run(Runner.java:509)
    at org.eclipse.jetty.runner.Runner.main(Runner.java:557)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at WarMain.launchWebServer(WarMain.java:187)
    at WarMain.start(WarMain.java:345)
    at JarMain.doStart(JarMain.java:233)
    at WarMain.main(WarMain.java:367)

any tips on how to add logging without getting in the way of startup jitting?

fyi my rails app uses the following rack middleware:

class Application < Rails::Application

  # ...

  config.middleware.delete(Rails::Rack::Logger)

  config.middleware.insert_before(Rack::Sendfile, Rack::Deflater)

  config.middleware.insert_after(
    Rack::Runtime,
    Prometheus::Client::Rack::Collector)

  config.middleware.insert_after(
    Prometheus::Client::Rack::Collector,
    Prometheus::Client::Rack::Exporter)

  # ...

end

the prometheus middleware comes from gem 'prometheus-client', '0.6.0'

i've seen some errors in production related to the prometheus exporter. this is a bit of a wild guess, but you may be able to reproduce locally by configuring your rails app likewise

@headius
Copy link
Member

headius commented Nov 8, 2016

Ok, let's look at the errors you have now.

2016-11-02T15:17:08.571-07:00 [main] INFO JITCompiler : Could not compile; passes run: [org.jruby.ir.passes.LocalOptimizationPass@584ed071, org.jruby.ir.passes.DeadCodeElimination@805d30ee, org.jruby.ir.passes.OptimizeDynScopesPass@f61f6f63, org.jruby.ir.passes.OptimizeDelegationPass@6c7a3ce9, org.jruby.ir.passes.AddCallProtocolInstructions@87f2d916, org.jruby.ir.passes.AddMissingInitsPass@a7c39fef]: <block> uri:classloader:/gems/phony-2.15.26/lib/phony/countries.rb_CLOSURE_1.uri:classloader:/gems/phony-2.15.26/lib/phony/countries.rb_CLOSURE_1 at uri:classloader:/gems/phony-2.15.26/lib/phony/countries.rb:43 because of: "Could not compile org.jruby.runtime.MixedModeIRBlockBody@2cfc81f8; instruction count 2315 exceeds threshold of 2000"
org.jruby.compiler.NotCompilableException: Could not compile org.jruby.runtime.MixedModeIRBlockBody@2cfc81f8; instruction count 2315 exceeds threshold of 2000
    at org.jruby.compiler.JITCompiler$BlockJITClassGenerator.compile(JITCompiler.java:553)
    at org.jruby.compiler.JITCompiler$BlockJITTask.run(JITCompiler.java:362)
    at org.jruby.compiler.JITCompiler.buildThresholdReached(JITCompiler.java:196)

This one is expected. We currently have fairly conservative size limits for jitting code to JVM bytecode, since too-large code will remain in the JVM's interpreter, running slower than our interpreters.

2016-11-02T15:33:56.940-07:00 [pool-1-thread-3] INFO JITCompiler : Could not compile; passes run: [org.jruby.ir.passes.LocalOptimizationPass@584ed071, org.jruby.ir.passes.DeadCodeElimination@805d30ee, org.jruby.ir.passes.OptimizeDynScopesPass@f61f6f63, org.jruby.ir.passes.OptimizeDelegationPass@6c7a3ce9, org.jruby.ir.passes.AddCallProtocolInstructions@87f2d916, org.jruby.ir.passes.AddMissingInitsPass@a7c39fef]: HashWithIndifferentAccess ActiveSupport::HashWithIndifferentAccess.to_hash at uri:classloader:/gems/activesupport-4.2.0/lib/active_support/hash_with_indifferent_access.rb:248 because of: "null"
java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
    at java.util.ArrayList$Itr.next(ArrayList.java:851)
    at org.jruby.ir.IRScope.setUpUseDefLocalVarMaps(IRScope.java:942)
    at org.jruby.ir.dataflow.analyses.DefinedVariableNode.identifyUndefinedVarsInClosure(DefinedVariableNode.java:91)
    at org.jruby.ir.dataflow.analyses.DefinedVariableNode.identifyInits(DefinedVariableNode.java:145)
    at org.jruby.ir.dataflow.analyses.DefinedVariablesProblem.findUndefinedVars(DefinedVariablesProblem.java:45)

This one is a real problem that should be filed separately and fixed. It relates to one of our optimization passes in JRuby's compiler. Shouldn't be hard to fix. cc @subbuss @enebo

i tried overriding Rack::Builder.initialize to add logging (this is the method defined at rack/lib/builder.rb:55). unfortunately, the monkey patch triggers a new error:

java.lang.NullPointerException
    at org.jruby.runtime.backtrace.RubyStackTraceElement.mriStyleString(RubyStackTraceElement.java:77)
    at org.jruby.runtime.ThreadContext.createCallerBacktrace(ThreadContext.java:699)
    at org.jruby.RubyKernel.caller20(RubyKernel.java:1063)
    at org.jruby.RubyKernel$INVOKER$s$0$2$caller20.call(RubyKernel$INVOKER$s$0$2$caller20.gen)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:195)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.rack.builder.invokeOther4:caller(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/rack/builder.rb:7)
    at Users.fedenusy.dev.pocketz_minus_rails.web.WEB_minus_INF.common.lib.gem_ext.rack.builder.RUBY$method$initialize$0(/Users/fedenusy/dev/pocketz-rails/web/WEB-INF/common/lib/gem_ext/rack/builder.rb:7)

This also looks like a problem with how we generate the "caller" backtrace. Judging by the line number indicated, it looks like something from the original Java stack trace element is null and we're not guarding for that. I can make a quick fix for this.

So, please file the second item above as a separate issue and I'll fix the remaining "caller" problem.

@headius headius closed this as completed Nov 8, 2016
headius added a commit that referenced this issue Nov 8, 2016
@headius
Ensure fileName is never null. See #4255
48cc9ba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
jit JRuby 9000 needs tests regression
Projects
None yet
Milestone
JRuby 9.1.6.0
Development

No branches or pull requests
2 participants
@headius @fedenusy