Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jruby crashes when reading large file #4704

Open
kirillrdy opened this issue Jul 2, 2017 · 4 comments
Open

jruby crashes when reading large file #4704

kirillrdy opened this issue Jul 2, 2017 · 4 comments

Comments

@kirillrdy
Copy link

kirillrdy commented Jul 2, 2017 •
edited
Loading

Environment

starting jruby 9.0.3.0, crash when reading large file

code

dd if=/dev/zero of=large_file count=1 bs=1400M
File.open("large_file", "rb") do |f|
 f.read
end

Results in following stack trace:

Java::JavaLang::NegativeArraySizeException: 
	from org.jruby.util.ByteList.ensure(ByteList.java:341)
	from org.jruby.RubyString.modify(RubyString.java:950)
	from org.jruby.RubyString.modifyExpand(RubyString.java:960)
	from org.jruby.util.io.EncodingUtils.setStrBuf(EncodingUtils.java:1116)
	from org.jruby.util.io.OpenFile.fread(OpenFile.java:1755)
	from org.jruby.util.io.OpenFile.readAll(OpenFile.java:1669)
	from org.jruby.RubyIO.read(RubyIO.java:3009)
	from org.jruby.RubyIO.read(RubyIO.java:2987)
	from org.jruby.RubyIO$INVOKER$i$read.call(RubyIO$INVOKER$i$read.gen)
	from org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
	from org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
	from org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
	from org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
	from org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
	from org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
	from org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)
... 122 levels...
	from org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
	from org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
	from org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
	from org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
	from home.kirillvr.jruby.bin.irb.invokeOther3:start(/home/kirillvr/jruby/bin/irb:13)
	from home.kirillvr.jruby.bin.irb.RUBY$script(/home/kirillvr/jruby/bin/irb:13)
	from java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:625)
	from org.jruby.ir.Compiler$1.load(Compiler.java:95)
	from org.jruby.Ruby.runScript(Ruby.java:827)
	from org.jruby.Ruby.runNormally(Ruby.java:746)
	from org.jruby.Ruby.runNormally(Ruby.java:764)
	from org.jruby.Ruby.runFromMain(Ruby.java:577)
	from org.jruby.Main.doRunFromMain(Main.java:417)
	from org.jruby.Main.internalRun(Main.java:305)
	from org.jruby.Main.run(Main.java:232)
	from org.jruby.Main.main(Main.java:204)

Provide at least:

  • JRuby version (jruby -v) and command line (flags, JRUBY_OPTS, etc)
    jruby 9.1.11.0-SNAPSHOT (2.3.3) 2017-06-14 83098f6 Java HotSpot(TM) 64-Bit Server VM 25.51-b03 on 1.8.0_51-b16 +jit [linux-x86_64]
    JRUBY_OPTS="-J-server -J-Xmx10g -J-Xmn1024m -J-Xms2048m -J-XX:+UseG1GC"
  • Operating system and platform (e.g. uname -a)
    Linux tsugaike 4.11.7-300.fc26.x86_64 break script engine #1 SMP Mon Jun 26 14:54:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Other relevant info you may wish to add:

  • Installed or activated gems
  • Application/framework version (e.g. Rails, Sinatra)
  • Environment variables

Expected Behavior

Its possibly related to #3792
MRI can deal with strings that length.
So either jruby string length has to be int size and we are missing string bounds check,
or its a jruby bug because string that length can be read in mri
@tlxxzj
Copy link

tlxxzj commented Aug 23, 2017

@enebo
in ByteList.java

    public void ensure(int length) {
        if (begin + length > bytes.length) {
            byte[] tmp = new byte[Math.min(Integer.MAX_VALUE, length + (length >>> 1))];
            System.arraycopy(bytes, begin, tmp, 0, realSize);
            bytes = tmp;
            begin = 0;
            invalidate();
        }
    }

length + (length >>> 1) may overflow
1500000000 + (1500000000 >>> 1) = -2044967296

@kirillrdy
Copy link
Author

Thanks @tlxxzj for update

@glurp glurp mentioned this issue Sep 8, 2017
Closed
@CodingAnarchy
Copy link

CodingAnarchy commented Oct 4, 2019 •
edited
Loading

MRI uses a C long data type for the lengths involved here, which on most 64-bit systems will be a 64-bit signed integer. JRuby appears to use Java's int type in these places, which I believe is a 32-bit signed integer regardless of the underlying infrastructure.

Comparing JRuby's Array#join implementation with MRI's implementation indicates they are nearly identical functionally otherwise.

If this is the cause of the disparity, I would expect that replacing int data types with long in the Array and ByteList implementations would resolve this issue (though I fear I am not familiar enough with Java or JRuby to do this safely), and I would also expect MRI to break similarly on a 32-bit system (or a 64-bit system that does not widen the long type).

@kares
Copy link
Member

kares commented Oct 5, 2019

you can not create an array of a long length in Java.
so this would need much more eork than replacing an int field with a long one.

JuicyDragon added a commit to Nuix/Investigator-Report that referenced this issue Jul 7, 2020
@JuicyDragon
Updated how text is written to item details HTML files
8152de3 
Text is now written to item details files directly rather than being part of template rendering in an attempt to prevent larger content text values from throwing the JRuby error described here: jruby/jruby#4704
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kirillrdy @kares @CodingAnarchy @tlxxzj