interesting have you tried triggering a full GC - maybe those classes would end up being collected ?
anyways, this seems most likely like a JIT regression ... // cc @headius

Just tried to run GC on two different servers, it didn't remove any classes.

the tmp prefix (and the fact that based on the screenshot the classes are empty) seems to indicate that these aren't JRuby's JIT-ed classes nor any of the (generated) internal ones - they mostly have "rubyjit" or "rubyobj" package prefix.

The classes are not empty. They have static fields. Examples:

https://www.dropbox.com/s/xn4jx8e8yy0oh3l/Screenshot%202016-09-08%2011.08.27.png?dl=0
https://www.dropbox.com/s/ua7j63l59l33iwu/Screenshot%202016-09-08%2011.08.37.png?dl=0
https://www.dropbox.com/s/da1uvg574fepin3/Screenshot%202016-09-08%2011.08.42.png?dl=0
https://www.dropbox.com/s/0bwum446x6zvlts/Screenshot%202016-09-08%2011.08.54.png?dl=0

And btw. there are no class with "rubyjit" or "rubyobj" package prefix in the dump.

Would it help if I upload the dump somewhere?

ah thanks - that confirms it's coming from the JITCompiler ... seems it's not using the "rubyjit" prefix now.
they might be "valid" (used) and it makes sense there's more of them over time than (probably more stuff JIT compiling in 9.1 than in 9.0). each has it's own little class-loader but contains different data.

it would make sense to have a better name for them so they do not seem that much similar ...

they might be nothing to worry much, how long did you let it roll ? did you run into any issues ?

With all other settings unchanged our JRuby processes grow to consume a few hundred MB more memory now, which lead to crashing servers. That's when I started to investigate and look for things that changed since the upgrade.

On our main project I have seen up to 3x as many classes compared to 9.0 before the server ran out of RAM. I don't know how far it would go. The used Metaspace is increasing linear with the class count. Is there a limit to how many classes the JITCompiler will create? What would happen if I limit the Metaspace with the -XX:MaxMetaspaceSize option?

thanks - knowing there's a crash eventually is important ... we've seen a report about increased usage #4127 but turned out app specific (no clear JRuby leakage found looking at the heap-dump). so maybe there's something after all. if possible you could try 9.1.2.0 to confirm whether it's also acting the same.

thread dumps would be useful at some point - but if there's not a clear leak candidate(s) it might be tricky.

The used Metaspace is increasing linear with the class count. Is there a limit to how many classes the JITCompiler will create?

there's -Xjit.max but it might not be taken into consideration on 9K - if you do see linear increases than there probably is an issue. we'll eventually need to look at heap dumps (it would be best to compare with 9.1.2 if you're going to try and it works)

What would happen if I limit the Metaspace with the -XX:MaxMetaspaceSize option?

you should get an OutOfMemoryError eventually

This looks really weird...there shouldn't be more than a couple classes for any given jitted method, and usually just one. I'll have a look at your heap dump and see if I can figure out why we're duplicating so much code.

@Helle Would it be possible to get a copy of one of the dumps? If you're concerned about sensitive information, perhaps we could at least be online at the same time and walk through the heap together? I need to gather more information about these classes to understand why there's so many of them.

Is this running with a single JRuby instance or multiple instances?

It is certainly possible we're jitting more in 9.1 than in 9.0...many JIT bugs were fixed. But I still wouldn't expect to see so many duplicates.

Ahh another flag/property you could set is jruby.jit.logging=true (-Xjit.logging=true). Ideally over time that should show us that we actually are jitting methods multiple times.

@Helle How do u get class count in New Relic?

@headius from some of the images its apparent that those aren't the same thing (methods).
currently all methods/bodies from the same file seems to use (mangled) __FILE__ as the class name.

@headius There are two different projects showing this behaviour. Both running on two Servers with a single JRuby instance each. I will send you an email with links to the heap dumps. I generated the heap dumps from the smaller of the two projects with a night of very light traffic in between.

I am going to activate jit.logging=true on one of the servers and i'm gonna change my Xmx settings so the Metaspace has more room to grow. Let's see what happens over the weekend. As I'm on vacation next week, my colleague @cyrez will post the results.

@kares 9.1.2.0 has the same behaviour. It may be normal behaviour in 9.1 to JIT a lot more, I noticed the difference because it made my servers run out of RAM...

@lephyrius I'm using the New Relic Java Agent for that

... also for comparison it might be valuable to have heap dumps from the same app using 9.0.5 as well

Looking forward to the results! And yes, heap dump would be useful. You can pass -XX:+HeapDumpOnOutOfMemoryError to the JVM and it will dump right before it croaks. Otherwise, just pull off a heap dump using the jmap command at some point when you're pretty sure the leaking is obvious.

Oops, got my bugs crossed...there's already a heap dump for this one. Yes, having a comparison dump for 9.0.5.0 would be great help.

Had some problems collecting useful data. I hope i can send you the dumps tomorrow, just deployed the application with 9.0.5.0 again. I will include a log file with Xjit.logging enabled from the weekend.

In my tests from yesterday it looks like our applications leaking in 9.0.5.0 also. So maybe it's a problem in our application. I will in investigate further and come back next week with more information.

https://www.dropbox.com/s/izznts4dqyk7grn/Screen%20Shot%202016-09-15%20at%2016.26.53.png?dl=0

Ok, keep us posted. We'll close this for now.