Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JVM Internal Errors/Segfaults under load #1024

Closed
sclasen opened this issue Sep 18, 2013 · 12 comments
Closed

JVM Internal Errors/Segfaults under load #1024

sclasen opened this issue Sep 18, 2013 · 12 comments

Comments

@sclasen
Copy link
Contributor

sclasen commented Sep 18, 2013

The problem is always instanceKlass::remove_dependent_nmethod(nmethod*)

This is on jruby-1.7.4

EDIT: Looks like I had a indy enabled via JRUBY_OPTS but the rest is accurate

Only seems to happen under load testing that lasts for a decent amount of time. A 5 min test will not crash, but a one hour test will.

Reproduced on both

OpenJDK

JRE version: 7.0_25-b30
Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)

and

Oracle Server JDK

JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 compressed oops)

Errors are either

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (instanceKlass.cpp:1534), pid=11491, tid=139819998455552
#  Error: ShouldNotReachHere()
#

or

#  SIGSEGV (0xb) at pc=0x00007f59bec82b91, pid=12846, tid=140023237494528
#
# JRE version: 7.0_25-b30
# Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x50db91]  instanceKlass::remove_dependent_nmethod(nmethod*)+0x1

A sanitized gist of one hs error log is here https://gist.github.com/sclasen/d2e5b413a7107bcbfeb5

LMK what more information I can provide

@chrisseaton
Copy link
Contributor

Sorry nobody has gotten back to you. Does this problem still stand? If you'd like us to investigate can you give use a test case where we can reproduce the problem?

@rtoma
Copy link

rtoma commented May 1, 2015

Got bitten by this issue too on Openjdk 7.0_75-b13 while running Logstash 1.1.13 (jruby 1.8/1.9?):

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (instanceKlass.cpp:1543), pid=23652, tid=140690659669760
#  Error: ShouldNotReachHere()
#
# JRE version: OpenJDK Runtime Environment (7.0_75-b13) (build 1.7.0_75-mockbuild_2015_01_08_20_32-b00)
# Java VM: OpenJDK 64-Bit Server VM (24.75-b04 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.5.4
# Distribution: Built on Red Hat Enterprise Linux Server release 6.6 (Santiago) (Thu Jan  8 20:32:29 EST 2015)

@ratnikov
Copy link
Contributor

I've run into this in our very custom setup. There's a workaround of turning Method flushing off with -XX:-MethodFlushing. This may increase Code Cache memory consumption since JITTed methods won't get reclaimed so if you'd like to give it a try I'd recommend making sure you're not hitting those limits (although the risk should be pretty low: essentially no more JIT).

@headius
Copy link
Member

headius commented May 11, 2016

Much too old, workarounds provided, and almost certainly stale. We'll just call it invalid. Any new crashers you see, please file new issues.

@headius headius closed this as completed May 11, 2016
@headius headius added this to the Invalid or Duplicate milestone May 11, 2016
@ratnikov
Copy link
Contributor

headius@, I can reliably reproduce this issue with 1.7.18. Of course that's a bit stale, but I can see if I can reproduce it with later versions of JRuby too. Maybe give me 2 weeks to see and if still no positive, then obsolete it?

@headius
Copy link
Member

headius commented May 13, 2016

@ratnikov I'd be more interested to know if you can reproduce it with recent Java 7 or 8 builds. Both of the versions you list are rather old.

If we have a reproducible case on a current Java 8, we can take this to Oracle folks to investigate.

So yes, get us something we can reproduce with latest JRuby 1.7/9k and recent JDKs, and we'll try to escalate!

@headius
Copy link
Member

headius commented May 13, 2016

To clarify: if you can reproduce with latest JRuby versions on recent JDK versions, we can go ahead and reopen this.

@ratnikov
Copy link
Contributor

will do. This is what I have so far:

  1. Can reproduce on 1.7.8 (I was wrong about 18).
  2. 1.7 JRE, but special Google build (it's mostly based on openjdk but may have slight deviations).
  3. Only with CMS collector, seems to work fine with ParallelGC.
  4. Definitely is due to JRuby's JITing (the lower JIT threshold the more reliable the breakage is, and if I turn JIT off, cannot reproduce it anymore).

Definitely need to try with latest 1.7 version to see if maybe the issue went away.

@ratnikov
Copy link
Contributor

ratnikov commented May 13, 2016

And correction: 1.8 JRE. Here's one of the crashes I get:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f72263c4ef8, pid=508502, tid=508548
#
# JRE version: OpenJDK Runtime Environment (8.0) (build 1.8.0-google-v7-106887034-105643046)
# Java VM: OpenJDK 64-Bit Server VM (25.51-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x66bef8]  InstanceKlass::remove_dependent_nmethod(nmethod*)+0x8
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid508502.log
Compiled method (c2)   71853 3216             rubyjit.HashWithIndifferentAccess$$convert_key_74DAAF660E3A7128DBA2AAD20F154BF1839C085C1073862849::__file__ (42 bytes)
 total in heap  [0x00007f721a8f17d0,0x00007f721a8f3270] = 6816
 relocation     [0x00007f721a8f18f8,0x00007f721a8f19d0] = 216
 main code      [0x00007f721a8f19e0,0x00007f721a8f1f80] = 1440
 stub code      [0x00007f721a8f1f80,0x00007f721a8f1ff0] = 112
 oops           [0x00007f721a8f1ff0,0x00007f721a8f2128] = 312
 metadata       [0x00007f721a8f2128,0x00007f721a8f22d0] = 424
 scopes data    [0x00007f721a8f22d0,0x00007f721a8f2ac8] = 2040
 scopes pcs     [0x00007f721a8f2ac8,0x00007f721a8f3118] = 1616
 dependencies   [0x00007f721a8f3118,0x00007f721a8f31a0] = 136
 handler table  [0x00007f721a8f31a0,0x00007f721a8f3218] = 120
 nul chk table  [0x00007f721a8f3218,0x00007f721a8f3270] = 88
Compiled method (c2)   71853 3216             rubyjit.HashWithIndifferentAccess$$convert_key_74DAAF660E3A7128DBA2AAD20F154BF1839C085C1073862849::__file__ (42 bytes)
 total in heap  [0x00007f721a8f17d0,0x00007f721a8f3270] = 6816
 relocation     [0x00007f721a8f18f8,0x00007f721a8f19d0] = 216
 main code      [0x00007f721a8f19e0,0x00007f721a8f1f80] = 1440
 stub code      [0x00007f721a8f1f80,0x00007f721a8f1ff0] = 112
 oops           [0x00007f721a8f1ff0,0x00007f721a8f2128] = 312
 metadata       [0x00007f721a8f2128,0x00007f721a8f22d0] = 424
 scopes data    [0x00007f721a8f22d0,0x00007f721a8f2ac8] = 2040
 scopes pcs     [0x00007f721a8f2ac8,0x00007f721a8f3118] = 1616
 dependencies   [0x00007f721a8f3118,0x00007f721a8f31a0] = 136
 handler table  [0x00007f721a8f31a0,0x00007f721a8f3218] = 120
 nul chk table  [0x00007f721a8f3218,0x00007f721a8f3270] = 88

@headius
Copy link
Member

headius commented May 13, 2016

Some issues that look related:

https://bugs.openjdk.java.net/browse/JDK-8139595
https://bugs.openjdk.java.net/browse/JDK-8129810

The method crashing is not particularly odd:

      def convert_key(key)
        key.kind_of?(Symbol) ? key.to_s : key
      end

There's nothing here that our JIT or the JVM JIT should trip over.

I will say it's probably likely that we won't see this in 9k because it's a completely rewritten JIT with very different output.

@ratnikov
Copy link
Contributor

Just verified that this seems to be fixed with JRuby 1.7.10 and 1.7.25, so likely it was fixed in 1.7.9 or 1.7.10 and is obsolete.

@headius
Copy link
Member

headius commented May 16, 2016

Ok thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants