@headius here's the full backtrace as requested:

• https://gist.github.com/digitalextremist/830d649bc2f65e6261c47eafa12578a9

@digitalextremist Ok, yeah...the exception seems to be raised by someone else via our thread eventing. That's harder to investigate since we don't have logging of cross-thread exception raises right now.

The exception itself looks like it's probably the anonymous exception Timeout throws.

It's unusual to me that it's bubbling out of a yield. Timing out in yield would mean that fiber A resumes fiber B, fiber B yields back to A, and A times out and bubbles this exception back to B. That's an unusual direction for exceptions to flow...generally they only come out of resume, not out of yield.

Ok, in order to continue on this I think we need a reproduction, or @digitalextremist will need to do some debugging locally.

A patch to start investigating this would be to add logging to ThreadFiber.exchangeWithFiber's catch block to print out all exceptions. I suspect it is propagating something the wrong direction.

I remember there used to be a Thread-based mode for Celluloid too...perhaps it still exists and we can try it?

Another place to debug would be RubyThread.receiveMail. This method is called when thread A wants to deliver a kill or raise event to thread B, and it runs in the context of thread A. We could see who is triggering this cross-thread (cross-fiber) exception then.

@headius I was able to work around this for now by cutting out Timeout.timeout and using the HTTP timeouts :per_operation which I believe triggers a timeout on a socket-level, rather than using the infamous Timeout API.

Either way I remember the situation clearly and will try to come back with a repro that works. I think the key here in reproducing it reliably would be to have a TCPServer in the repro itself that opens a connection but then does nothing, so there's not an immediate error of connection refused or similar.

Thank you very much for looking through this. Our interactions caused me to think of the workaround, brought up things in Celluloid that are fruitful. Like I said, I'm definitely down to ship beer :)

Still a bit of a mystery and unlikely to be investigated for the quick 9.1.1 release. Punting.

@headius this resurfaced with Rubinius, except the class did bubble up in that case.

In that incarnation, it appears to be Mutex#synchronize being bombarded... timing out at this specific line:

• https://github.com/celluloid/celluloid-essentials/blob/master/lib/celluloid/internals/thread_handle.rb#L16

Which makes sense, because here is what complains:

• https://github.com/celluloid/celluloid/blob/master/lib/celluloid/group/spawner.rb#L52

Perhaps I missed it, but what was the actual exception class that bubbled out?

Pardon me, it's Timeout::Error

Ok...so is this just getting through because nobody expects to rescue it? Is there anything we need to fix on the JRuby side? I assume since the same issue appeared on Rubinius, it may be an exception-handling gap in Celluloid...

I'm troubleshooting the Celluloid end -- but the fact that the class being bounced on the jRuby side is an anonymous class still persists; although yes, something is falling through on the Celluloid side.

Ok, I can at least investigate why the class is anonymous.

It looks like the anonymous exception code is very old...no longer present in MRI's timeout.rb after August 2013. I will do a pass over timeout.rb and our version and try to ensure they match.

Ok, so here's the deal.

If you provide a specific exception type to Timeout::timout, it will raise a new instance of that exception when the timer runs out. If you do not provide an error. It eventually raises Timeout::Error. I say eventually, because MRI's timeout.rb has used various other exceptions to actually unroll the timeout-protected code back to the code in timeout.rb that decides how to propagate the exception.

At one point, it did Class.new(Exception) every time you called for a timeout guard. This is the anonymous exception we emulate currently (inside an exception type called AnonymousException but with its name removed to match MRI.

Later, they realized the high cost of creating a new class every time and instead created a single ExitException type they used to unroll.

And most recently it appears they now simply raise Timeout::Error right away but post-process it a bit to scrub out the backtrace lines related to Timeout::timeout.

The original intent was that this should be an exception users never see and never rescue so it propagates all the way out to the timeout call.

Now, the fact that you saw this exception makes me think we're not properly handling it when it bubbles out, and it's continuing to propagate into user code. That could be a bug, or more likely it's a problem with the asynchronous nature of Timeout causing the anon exception to be raised outside the timeout call.

I will do some experimenting and see if I can get this to reproduce based on my theory.

I have pushed a new_timeout branch to my fork (which reverts us back to using MRI's timeout.rb) and opened #3895 to test it out. If you get a chance, you could try to reproduce your issues on a build from that branch.

I doubt will stick with unmodified timeout.rb since it has a few big performance issues, but this will at least let us know how well it integrates into the rest of the system.

Workarounds include not using timeout and using it with an explicit exception, so I think we can punt any remaining rewrite of timeout.rb (or re-incorporation of MRI's version, with bottlenecks fixed) to 9.1.4.0.

I've pushed an updated implementation of Timeout that shares more code with MRI and always uses the same Timeout::Error exception rather than our weird anonymous one. I believe this will resolve this issue.