File.seek fails on large file #3435

jegt · 2015-10-30T11:56:34Z

File.seek fails on files 2GB or larger on Solaris and Linux.

Test program:
[2147483647, 2147483648].each do |position|
file = File.open('test.tar', 'rb')
file.seek(position)
puts "seek to #{position} succeeded"
end

output:
seek to 2147483647 succeeded
Errno::EBADF: Bad file number - test.tar
seek at org/jruby/RubyIO.java:1648
block in seek.rb at seek.rb:3
each at org/jruby/RubyArray.java:1560
at seek.rb:1

jruby 9.0.3.0 (2.2.2) 2015-10-21 633c9aa Java HotSpot(TM) 64-Bit Server VM 25.65-b01 on 1.8.0_65-b17 +jit [SunOS-amd64]

SunOS org00 5.11 omnios-f090f73 i86pc i386 i86pc

jruby 9.0.3.0 (2.2.2) 2015-10-21 633c9aa OpenJDK 64-Bit Server VM 25.45-b02 on 1.8.0_45-internal-b14 +jit [linux-amd64]

Linux worker04 3.13.0-67-generic #110-Ubuntu SMP Fri Oct 23 13:24:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I discovered this when trying to use multipart uploads to S3 using the official aws-sdk gem.

When using the gem i got mostly Errno::EAGAIN and sometimes Errno::ENOENT. The test program seems to always fail with Errno::EBADF.

The same test program works fine on MRI: ruby 2.1.3p242 (2014-09-19 revision 47630) [i386-solaris2.11]

The text was updated successfully, but these errors were encountered:

gucki · 2016-01-22T09:56:48Z

The same happens when using tell on a file where the position is > 2 GB.

irb(main):024:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>

irb(main):030:0> f.tell
=> 0

irb(main):025:0> puts f.size
4294967296
=> nil

irb(main):026:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):26:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):027:0> puts f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):27:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):028:0> f.close
=> nil

I tried with latest ubuntu 64-bit openjdk and

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

gucki · 2016-01-22T10:09:17Z

It seems like this is a regression of 9.0.4, as it works with 1.7.21:

jruby 1.7.21 (1.9.3p551) 2015-07-07 a741a82 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
=> 0
irb(main):005:0> f.tell
=> 2147483648
irb(main):006:0> f.seek(3*1024*1024*1024)
=> 0
irb(main):007:0> f.tell
=> 3221225472
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit

jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa Java HotSpot(TM) 64-Bit Server VM 25.66-b17 on 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):4:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):005:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):5:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):006:0> f.seek(3*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):6:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):007:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):7:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit

gucki · 2016-01-22T10:11:56Z

@headius Would be great to have this fixed in 9.0.5 👍

gucki · 2016-03-04T10:25:31Z

Is there anything we could do to get this fixed, liking sponsoring the fix?

enebo · 2016-03-07T16:07:01Z

This almost sounds like we switch IO pos from a long to an int? Marking for 9.1.0.0.

headius · 2016-04-20T18:21:02Z

JRuby 9.1 is using a long for position in seek, but still failing. Looking into it.

headius · 2016-04-20T18:44:21Z

Ok, so there's one small bug and one big bug here.

The small bug is that we were expecting all negative return values from tell and lseek to mean there was an error. This is incorrect; our IO#seek actually works fine, but the return value is the new position, and because our native lseek binding only has a 32-bit signed return value, this comes out negative.

The big bug is that our native lseek binding and many paths leading to it all use 32-bit int for the return value.

I'm fixing the first one, which makes seek and tell at least function properly at the native IO level:

[] ~/projects/jruby $ jruby -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
-2147483648

[] ~/projects/jruby $ ruby23 -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
2147483648

Note the second larger problem rears its head here with IO#tell being unable to return file positions larger than a signed 32-bit integer.

I'm going to mark this fixed, since at least the error is gone now and seek is doing what it should under the covers. I'll open a new bug for the larger issue.

headius · 2016-04-20T19:10:13Z

See #3817 for the larger issue.

gucki · 2016-04-20T19:49:45Z

@headius Thanks! 👍

jegt changed the title ~~File.seek fails on large file on Solaris~~ File.seek fails on large file Nov 7, 2015

kares added core JRuby 9000 regression labels Jan 22, 2016

enebo added this to the JRuby 9.1.0.0 milestone Mar 7, 2016

headius mentioned this issue Apr 20, 2016

IO#seek and tell do not properly handle offsets larger than int32 #3817

Closed

headius closed this as completed in 623bc59 Apr 20, 2016

headius mentioned this issue Apr 20, 2016

File#pos fails for files greater than java.lang.Integer::MAX_VALUE on 64-bit systems #3664

Closed

stickerbush mentioned this issue May 3, 2016

Aws::S3::MultipartUploadError when uploading large files aws/aws-sdk-ruby#1168

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Sponsors

File.seek fails on large file #3435

File.seek fails on large file #3435

jegt commented Oct 30, 2015

gucki commented Jan 22, 2016

gucki commented Jan 22, 2016

gucki commented Jan 22, 2016

gucki commented Mar 4, 2016

enebo commented Mar 7, 2016

headius commented Apr 20, 2016

headius commented Apr 20, 2016

headius commented Apr 20, 2016

gucki commented Apr 20, 2016

File.seek fails on large file #3435

File.seek fails on large file #3435

Comments

jegt commented Oct 30, 2015

gucki commented Jan 22, 2016

gucki commented Jan 22, 2016

gucki commented Jan 22, 2016

gucki commented Mar 4, 2016

enebo commented Mar 7, 2016

headius commented Apr 20, 2016

headius commented Apr 20, 2016

headius commented Apr 20, 2016

gucki commented Apr 20, 2016