Have any code to reproduce this? What do your JRUBY_OPTS and JAVA_OPTS look like?

I believe this is related to a case where AR will attempt (during exception raising) to to_s a model or query object (or something) that has an extremely large amount of data backing it up. The result is that it consumes too much memory and never gets to the point of raising the actual error.

I am not sure if we can solve this one. Perhaps giving more memory could help?

I'm closing this for now; if there is a reproduction and any fabulous idea to go on with, we'll revisit.

I'm seeing this same issue in my code right now. My code is handling images and other biggish files and it is quite common that when an exception is encountered the process consumes all the heap it has (3 Gigabytes currently) and then the Java heap space is thrown.

I very much suspect the same as headius. What about instead of just to_s on the object we could take only the first 1000 characters or something.

@mmustala: I'll drop this in the "ideas bucket." Consider me a crank until @headius weighs in:

Far from an ideal solution, but for objects that can become this large... I'd perhaps suggest monkey patching #to_s and using ObjectSpace.memsize_of(self) (require 'objspace' elsewhere). If it's very, very large then spit out a custom string and perhaps log the source of the issue.

For objects that can become really huge, overriding the behavior of #inspect is probably your only reliable course of action. Overriding methods is pretty standard once you start working with abnormally large blobs of data.

I'm guessing this is actually to blame for the memory explosion on exception raise: https://bugs.ruby-lang.org/issues/9725

Not really sure how to avoid this; it's a weird misfeature of Ruby that it inspects the target of a NameError in this way. Maybe anyone affected by this should +1 the issue I filed.

MRI has made no movement on the issue I filed, so I feel like we need to fix this unilaterally.

We can mitigate this problem by catching OutOfMemoryError in the NameError logic and falling back on a simple class name. That could capture OOME unrelated to the string, but it's very unlikely (and if it's really OOM, another one will come along shortly).

Beyond that, I think we should just modify NameError to not do this. @enebo?

The problem with catching OOMError is it's inherently unreliable. If thread A is doing something insane (like building a ginormous string for an exception) and then thread B does a reasonable allocation at the wrong time then B will get an OOM and probably die a horrible death.

I think your discussion on the MRI bug is spot on: you have to prevent the ginormous string allocation in the first place. Detecting the OOM after-the-fact will lead to even harder to diagnose problems, albeit more rarely.

Hard to say if MRI would accept this change, since it would have dual behavior across Error objects. It helps that in the land of MRI, memory swap is freeeee!

You should definitely solve this unilaterally. If it doesn't impact performance too hard, you could even juggle behavior based on the Obj's size... Or swap in a helpful String to let them know why there's no inspect output.

There's no bug here for us to fix...it's an unfortunate side effect of standard Ruby behavior.

JRuby 9.1 will not limit memory to 500MB, so perhaps it will rarely run into a NameError that's too big.

If you would like to see this changed, add your vote to the CRuby bug: https://bugs.ruby-lang.org/issues/9725

@headius Any suggestions for getting around this issue? Looks like we are facing this with JRuby 9.0.4.0 and Sidekiq.

I didn't find anything else but overriding the inspectmethod in the class that is causing the problem. That way you can make sure inspect is returning a string that is not too big.

Here is the stacktrace https://gist.github.com/prathamesh-sonpatki/0d6ce40f703a8ce930ff4b008af7002e

I'm going to weigh in on this...I don't have any direct experience with this bug but I have been conversing with someone on Stack Overflow who does -- he gave up using JRuby, even though he needed to call Java libraries and it was the obvious choice, because of this problem (see comments in http://stackoverflow.com/questions/37316055/file-write-happening-asynchronously/37317141?noredirect=1#comment62158497_37317141).

On the higher level, my opinion is that class authors should not presume to know how to_s and inspect are called, and support their use by the (Ruby or Java) VM and frameworks out of their field of influence by ensuring that the size of the strings they return is reasonable (maybe <= 1K?), and that the cost of calling them is low (e.g. not to read an entire DB table). I think we should be able to assume that we can call to_s or inspect to build a log message, for example, and it will not overpower the virtual machine. If a method that returns a large amount of data is needed, it can be implemented with a new name and called explicitly by those who have the specialized knowledge of it.

So I wonder if we're looking in the wrong place for the solution. If ActiveRecord includes reads in an entire table and includes its data in a returned string, IMO this is the problem.

If we assume that implementers of to_s and inspect should limit the size and cost of the returned strings, then it is reasonable to include them in an exception's text as is currently the case.

However, given the severity of the situation, I think Charlie's suggestion is a quicker solution, and one that is more reliable, since we don't know what classes will be coded in the future.

My vote would be to fix this ASAP, even if it means that JRuby deviates slightly from MRI.

@keithrbennett are you sure that was the link you intended to include? I do not understand how it is related.

As a general statement I find this to be bugs in various libraries who dump the planet. If you are going to dump a DB as output then perhaps you need to limit output? As a broader statement it also shows how Ruby might have been better off using IO-based API for this (e.g. output(print_io=$stdout)).

I have a different proposal for to_s on NameError. I think arguably this is something where some people will want full output and others will not. We can make a new require:

require 'abbreviated_to_s'

class NameError
   include AbbreviatedToS
end

By including this to NameError we will check to see if it is included and then do a respond_to? :abbreviated_to_s and call that instead.

This will put a possible burden on library author to fix the general issue by making their own abbreviated output. I think though if you plan on not including everything in output it is probably important that library auhors can make a best effort on what is displayed.

A generic implementation can be made as well. It would need to be somewhat reflective and inspect sizes of various ivars and their values and elide values which are too big. So this could be something like:

require 'abbreviated_to_s/reflective'

class ActiveRecord::Base
   include AbbreviatedToS::Reflective
end

Then if people run into this they can add a little magic in a Rails initializer and the problem is fixed.

So on our side we would need NameError (or anything else which would need this) see if the module in question is in its hierarchy and call it instead of to_s if so. Someone would need to write these small shims and I think export them as a gem.

This also feels like it could be baked into JRuby and MRI as well but perhaps just trying to solve this in one impl first might be a good path forward?

Note: abbreviated_to_s and the names I picked are just stakeholder names. I do not really care what name is used so long as people like a name.

@tenderlove @sgrif Might either of you know if this sort of problem has been fixed in ActiveRecord? To be more specific: a NameError raised against a model with a lot of relationships might drag along a LOT of output when the NameError is eventually printed out.

See also https://bugs.ruby-lang.org/issues/9725 which has stalled.

I don't believe so, no. I'm not sure if there's much that we can do on the AR side short of re-implementing NameError

I may be misunderstanding this issue, but...

@sgrif Is it true that ActiveRecord's to_s is pulling in an entire table's data into the string? If this is so, then I would argue that the issue is not that NameError is calling to_s, but rather that the to_s is building and returning such a huge string to begin with.

I think the higher level question that the community needs to answer is "given the havoc they can wreak, do we want to condone and support huge to_s strings?".

In my opinion, the answer is no, because I think the value of enabling any code to be able to call to_s or inspect on unknown objects without fear of bringing down the system is greater than the value of the freedom to implement huge strings.

@enebo The JRuby discussion was buried in the comments. Search the page for "JRuby" and you'll see it. Also, I'm sure your solution is technically sound, you know way more than I about these things, but I still think that's way more complexity than the freedom to return huge strings is worth.

No we don't pull the entire table. On instances we actually shouldn't be grabbing anything from #inspect. On the class object we load the schema if it's not loaded. On a relation instance we will load the query that it represents.

It seems possible the issues people have had here are not a problem in recent/current Rails...wish we could get some confirmation from someone who has had the issue in the past.

I'd be happy to do more digging if you can point me at more details of what
they were doing that would load so much data

On Fri, May 20, 2016, 9:41 AM Charles Oliver Nutter <
notifications@github.com> wrote:

It seems possible the issues people have had here are not a problem in
recent/current Rails...wish we could get some confirmation from someone who
has had the issue in the past.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#216 (comment)

I tried to create a simple app with the 9k jruby to demonstrate this error but couldn't get it reproduced. My original environment where the error happened is not available any more. I may give it another shot later if I find time for it.

I no longer work on the project where I originally experienced the issue in question. I believe the problem was that activerecord added each field of the record in the to_s.
The table in question had quite a big amount of fields and some of the fields contained large amounts of data.

I hope this helps.

Den 20. mai 2016 kl. 15.41 skrev Charles Oliver Nutter notifications@github.com:

It seems possible the issues people have had here are not a problem in recent/current Rails...wish we could get some confirmation from someone who has had the issue in the past.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub