there's been couple of sim.reports such as these lately but somehow always turned out to be app specific.
my first advise is to get a heap dump and start poking around or setup a JVM monitoring tool and monitor when memory usage increases. you should not need that much metaspace unless doing hot redeploys.

we can certainly look at heap dump but its time-consuming ;( and sometimes needs source code access.
... are you seeing the "leak-like" behaviour just lately, did you change something, does it get more load?
(all relevant to consider/think about if you're running into this just lately)

We did not have these issues with jruby-1.7.x before and the load did not increase really. We have about 30-40k rpm in average the last 7 days with peaks up to ~ 150k rpms. The heap dumps does not show anything specific. If i do a dump file now it has about 1 gb, even if htop tells about 11.3gb VIRT memory, and MAT tells me that its about 300 mb of data on heap. Which matches with the visual vm output.

Additional the Leak Suspect from MAT tells me:
20,790 instances of "org.jruby.ir.IRMethod", loaded by "" occupy 71,470.79 KB (21.69%) bytes.

8,804 instances of "org.jruby.MetaClass", loaded by "" occupy 64,254.32 KB (19.50%) bytes.

Biggest instances:

org.jruby.MetaClass @ 0x725658710 - 9,759.79 KB (2.96%) bytes.
org.jruby.MetaClass @ 0x72d22c4e8 - 6,511.81 KB (1.98%) bytes.

4,938 instances of "org.jruby.RubyClass", loaded by "" occupy 38,600.81 KB (11.72%) bytes.

Biggest instances:

org.jruby.RubyClass @ 0x700a7ebb0 - 4,300.45 KB (1.31%) bytes.

But as these "leak suspects" are part of the 300 mb on the heap, i does not expect them to be an issue.

thus if you're sure there's nothing crazy going on during requests can you let it (does it) fail with an OoME?
... that would confirm if there's a heap problem or one elsewhere.

The problem is, the system believes the process needs that much memory, but the java monitor shows a completely different status. When i restart the service (restart the process) the memory is freed again.

so it must be native memory than? just guessing - there's only so much one could do without looking into it.
naively, did you try going to 9.1.6.0 ?

It does sound like a native memory leak based on the numbers so far, especially if Java tools report a significantly smaller heap.

Yes I expect the same. Any clue how to find the leak? Any cool tools maybe?

@kares we did not try to use jruby 9.1.6.0 yet.

First thing would be to confirm 9.1.6.0 has the problem. There were many fixes in that release.

After that, there's a link on the wiki to flags for profiling memory, tools for getting and analyzing heap dumps, etc.

Oh, realized now some of those tools may not be super helpful for native memory leaks, and you already mentioned you've used MAT to analyze a heap dump. Ok...so let's get you on 9.1.6.0 and see if there's still a problem.

Hi
first sorry for the long silence.
Over the last weeks we worked around the leak by regular restarts of the app while also monitoring the leak during several jruby upgrades. Both 9.1.6.0 and 9.1.7.0 showed the leak.

Yesterday we performed the upgrade to 9.1.8.0 and as of now it looks like we're no longer leaking memory.

This shows the used system memory before and after the jruby upgrade.

@andreaseger Great news! Thanks for updating us.

Any idea which fix in 9.1.8.0 would have caused this? We are seeing a similar problem on some, but not all instances, where since moving to j9k we have unbounded native memory growth.

@the-michael-toy No we have no idea which fix caused this.

I think the issue has resurfaced in 9.1.12.0

74,318 instances of "org.jruby.RubyClass", loaded by "sun.misc.Launcher$AppClassLoader @ 0x70b600ab8" occupy 402,588,016 (23.82%) bytes.

Keywords
sun.misc.Launcher$AppClassLoader @ 0x70b600ab8
org.jruby.RubyClass

Here are the issues from MAT