Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JRuby Windows crash on org.jruby.ast.util.SexpMaker::process #4652

Open
GregWard opened this issue Jun 7, 2017 · 3 comments
Open

JRuby Windows crash on org.jruby.ast.util.SexpMaker::process #4652

GregWard opened this issue Jun 7, 2017 · 3 comments

Comments

@GregWard
Copy link

GregWard commented Jun 7, 2017 •
edited
Loading

Environment

Provide at least:

  • JRuby version - 1.7.26 (also attempted with 1.7.27)
  • Operating system - Windows Server 2012 R2

Running server jvm from JDK 8 build 131 64-bit
JAVA_OPTS FOR JVM:
-Dfile.encoding=UTF-8
-Djava.net.preferIPv4Stack=true
-XX:+HeapDumpOnOutOfMemoryError
-Djava.io.tmpdir=C:\Program Files\var\tmp
-Dnewrelic.config.file=C:\Program Files\New Relic\etc\newrelic-java-agent.yml
-Dnewrelic.config.log_file_path=C:\Program Files\New Relic\log
-Dnewrelic.environment=production

Expected Behavior

We use Jetty to run our rails server. We have been deploying on CentOS for years and have had little to no issues with JRuby. This is our first time work with Windows and expect it to work the same way.

Actual Behavior

We are seeing that when we occassionally load the server, it crashes (fairly consistently) on the c2 compiler thread on this method: org.jruby.ast.util.SexpMaker::process

We've reproduced the crash about 12 times now and each time this is the header that comes out:

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x74f63556, pid=3472, tid=0x000009bc

JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
Java VM: Java HotSpot(TM) Server VM (25.131-b11 mixed mode windows-x86 )
Problematic frame:
V [jvm.dll+0x363556]

Core dump written. Default location: C:\Program Files\bin\hs_err_pid3472.mdmp

--------------- T H R E A D ---------------

Current thread (0x50ab0000): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=2492, stack(0x51050000,0x510c0000)]

***YADDA YADDA YADDA ***

Current CompileTask:
C2: 79651 13529 4 org.jruby.ast.util.SexpMaker::process (93 bytes)

What we've tried:

We ran printcompilation and can see that it is always on this method, once it enters that optimized c2 compiling thread that it falls apart and we get the jvm crash.

We worked through, testing turning off compiling and found this solved the issue. We then ended up adding the JVM arg: -XX:CompileCommand=exclude,org.jruby.ast.util.SexpMaker::process

With this, we've done 150+ restarts consecutively of our server and have yet to see this issue arise again where the comparable without this arg was seeing it 1/5 restarts on average.

We also attempted to turn off Windows DEP on server 2012 but found this wasn't solving the problem. I've attached one of the hs_err logs and a replay log below as well.

replay_pid3472.txt
hs_err_pid3472.txt

PLEASE HELP!!
@headius
Copy link
Member

headius commented Jun 8, 2017

Wow, we haven't found a JVM crasher in a while 😁

So the first thing I should say is that in general, if the JVM crashes, we consider it to be the JVM's fault, and the JVM friends of JRuby agree with us. The JVM should not crash, or at least not crash in its own code as in this case. So this is likely a bug in this version of the "C2" JIT in the Hotspot JVM.

Are you running the Oracle JDK or some build of OpenJDK (possibly from Centos)?

You appear to be on the most recent version, in any case, so my first recommendation would be to try a different JVM. You can build it yourself or install various pre-built packages from e.g. Linux dists and companies like Azul (Zulu builds of OpenJDK).

My second recommendation would be to see if JRuby 9k crashes the same way, but we do still appear to use SexpMaker there as well. It also may not be trivial to get your JRuby 1.7 app migrated (but it would be good to make that move soon).

I'm glad you have a workaround! I'll poke around that method a bit and see if I can figure out the right flags to get better crash information from the JVM.

@headius
Copy link
Member

headius commented Jun 8, 2017

So the first thing I notice is that this method recurses. I'm wondering if the JVM JIT is walking off the end of the stack, and there's some stack protection missing in the C2 compiler at this point.

It's unclear how to set the stack size for a compiler thread. There's an option -XX:CompilerThreadStackSize=0 but I have not found proper use (and the default does not appear to reflect anything).

I would guess that disabling the C2 compiler with java flag -XX:TieredStopAtLevel=1 probably fixes the issue, but disables the better JIT. You might try testing with -XX:-TieredCompilation, which will use only C2 but might fail more reliably or not at all. Either would tell us something.

I'll ping the hotspot-compiler list and see if they have any thoughts.

@headius
Copy link
Member

headius commented Jun 8, 2017

Actually, before I ping the Hotspot folks I'll wait for you to try out other JVM builds and flags, maybe even recent JDK9, to see if anything else helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@headius @GregWard