Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues with Marshal.load #4683

Closed
slackfan opened this issue Jun 23, 2017 · 1 comment
Closed

Issues with Marshal.load #4683

slackfan opened this issue Jun 23, 2017 · 1 comment
Milestone
Invalid or Duplicate

Comments

@slackfan
Copy link

slackfan commented Jun 23, 2017 •
edited
Loading

Environment

  • JRuby version
jruby -v
jruby 9.1.12.0 (2.3.3) 2017-06-15 33c6439 Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +jit [mswin32-x86_64]
* JRuby version (`jruby -v`) and command line (flags, JRUBY_OPTS, etc)
  • Operating system: Any Windows, e.g. Microsoft Windows [Version 10.0.14393], 64bit.

Issue Description, Actual Behavior

I am using an internal rubygem server which mirrors rubygems.org plus additionally adds some own gems. While the Rubygem code tries to deserialize the Gemspec information (using Marshal.load()) from the server the JRuby JVM runs OOM if not defining a heap of 2GB. Using the Gemspec provided by Rubygems.org 256MB of heap are sufficient.

The obvious difference are about 100 additional Gems and that the serialized Gem specification is 3 time bigger (11mb instead of 37mb). The gzipped version of the gemspec (which is downloaded first) nearly has the exact same size (3.3mb vs 3.6mb)

The stack actually says, which is weird.

java.lang.OutOfMemoryError: Java heap space
	at org.joni.StackMachine.allocateStack(StackMachine.java:62)
	at org.joni.StackMachine.fetchStack(StackMachine.java:85)
	at org.joni.StackMachine.<init>(StackMachine.java:45)
	at org.joni.ByteCodeMachine.<init>(ByteCodeMachine.java:55)
	at org.joni.MatcherFactory$1.create(MatcherFactory.java:28)
	at org.joni.Regex.matcher(Regex.java:169)
	at org.jruby.RubyRegexp.search19(RubyRegexp.java:1219)
	at org.jruby.RubyRegexp.search19(RubyRegexp.java:1177)
	at org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1153)
	at org.jruby.RubyRegexp.op_match19(RubyRegexp.java:1092)
	at org.jruby.RubyString.op_match19(RubyString.java:1607)
	at org.jruby.RubyString$INVOKER$i$1$0$op_match19_DBG.call(RubyString$INVOKER$i$1$0$op_match19_DBG.gen)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:158)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
	at org.jruby.internal.runtime.methods.InterpretedIRMethod.INTERPRET_METHOD(InterpretedIRMethod.java:182)
	at org.jruby.internal.runtime.methods.InterpretedIRMethod.call(InterpretedIRMethod.java:173)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:158)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
	at org.jruby.internal.runtime.methods.InterpretedIRMethod.INTERPRET_METHOD(InterpretedIRMethod.java:182)
	at org.jruby.internal.runtime.methods.InterpretedIRMethod.call(InterpretedIRMethod.java:173)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:158)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
	at org.jruby.internal.runtime.methods.InterpretedIRMethod.INTERPRET_METHOD(InterpretedIRMethod.java:182)
	at org.jruby.internal.runtime.methods.InterpretedIRMethod.call(InterpretedIRMethod.java:173)
	at org.jruby.RubyClass.smartLoadNewUser(RubyClass.java:2049)
	at org.jruby.runtime.marshal.UnmarshalStream.userNewUnmarshal(UnmarshalStream.java:546)

Expected Behavior

I am seeking for guidance how to troubleshoot this. I started to analyse this via

ARGV.each do |file|
  spec_dump = IO.binread(file)
  results = Marshal.load(spec_dump)
  File.open("%s.log" % [file], 'wb') do |log| 
    log.write("%s %s %s \n" % [file, results.class, results.length])
    results.each do |result|
      log.write("%s\n" % [result.inspect])
    end
  end
end

The results do not explain the large behavioral differences. I also tried to get the ObjectSpace feature working described here https://github.com/jruby/jruby/wiki/PerformanceTuning but I did not get it working.

Can you recommend some more procedures helping me to understand the issue? In case you are interested, I uploaded the gemspec files gemspecs.zip
@slackfan
Copy link
Author

I found https://ilyabylich.svbtle.com/ruby-marshalling-from-a-to-z and used serialization to yaml to find the issue. The Gemserver provided gemspecs of all versions but not just the latest one. I close this ticket.

@enebo enebo added this to the Invalid or Duplicate milestone Sep 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Invalid or Duplicate
Development

No branches or pull requests
2 participants
@enebo @slackfan