Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encoding::UndefinedConversionError when IO.copy_streaming binary files #2779

Closed
janko opened this issue Mar 30, 2015 · 14 comments
Closed

Encoding::UndefinedConversionError when IO.copy_streaming binary files #2779

janko opened this issue Mar 30, 2015 · 14 comments

Comments

@janko
Copy link

janko commented Mar 30, 2015

We're trying to make Refile compatible with JRuby. We've almost made it (and it was fairly easy, I have to admit!), but we have one remaining bug.

In Refile we're using file wrappers, that is, we wrap an actual IO into a class which responds to #read, #eof?, #close and #size. This is all it takes to act as a real IO, because we can now pass them to IO.copy_stream, and it will work. However, when we wrap images (which are binary), IO.copy_stream raises an Encoding::UndefinedConversionError. The error doesn't happen if we pass the image IO directly, only when we use a wrapper (ActionDispatch::Http::UploadedFile or Refile::File, they both fail).

We weren't able to reproduce the problem (I posted an attempt of reproducing it), so we were hoping you could help us out. The pull request is refile/refile#187. I think you only need to run bundle install to set up the test suite, the branch is jruby on my fork.

@nirvdrum
Copy link
Contributor

Is this against JRuby 1.7.x, 9k, or both?

@janko
Copy link
Author

janko commented Mar 30, 2015

I tested against 9.0.0.0.rc1.

@nirvdrum
Copy link
Contributor

Do you mean 9.0.0.0.pre1? Or did you build from source? We haven't released rc1 yet.

@janko
Copy link
Author

janko commented Mar 30, 2015

Yes, sorry, that's the one I meant, 9.0.0.0.pre1.

@nirvdrum
Copy link
Contributor

Great. I can't say for certain that anything has been resolved, but a lot of encoding issues have been resolved since pre1. If you can test against jruby-head, that would be helpful. But pre2 should be releasing in the next couple weeks if you'd prefer to wait for a release.

@janko
Copy link
Author

janko commented Mar 30, 2015

Yeah, I guess it's not that important that we have this ASAP. Tried compiling JRuby from source, and did /path/to/bin/jruby -S bundle, but it cannot install Refile itself (which it shouldn't, because gemspec should make Bundler just take the Refile locally from the current directory). But it doesn't matter, we'll wait for the release.

@headius
Copy link
Member

headius commented Apr 2, 2015

Well your issue exposed a bug in autoload I have now fixed, but the branch still appears to try to activate the sqlite3 gem. Can you give me exact steps to reproduce the copy_stream issue?

@janko
Copy link
Author

janko commented Apr 3, 2015

Hmm, that's strange, I thought I set the Gemfile correctly, with JRuby's version of sqlite3. This setup worked for 9.0.0.0.pre1, but I think it also didn't for me on JRuby master. Maybe instead of

gem "sqlite3", platforms: [:ruby]
gem "activerecord-jdbcsqlite3-adapter", platforms: [:jruby]

it should be

unless RUBY_PLATFORM == "java"
  gem "sqlite3"
else
  gem "activerecord-jdbcsqlite3-adapter"
end

In the PR on Refile I explained what was happening and included a backtrace, and I included an isolated example I thought was going to reproduce the bug, but doesn't. You should be able to simply run bundle install and bundle exec rspec to run the test suite.

@headius
Copy link
Member

headius commented Apr 17, 2015

@janko-m I will try the alternate gemfile.

@headius
Copy link
Member

headius commented Apr 17, 2015

Seems like using sqlite3 in spec/refile/active_record_helper.rb causes the sqlite3 gem to be triggered, and I'm not sure how to stop that from happening. I did manage to bundle install everything ok.

@headius
Copy link
Member

headius commented Apr 17, 2015

@janko-m Please try to reduce your case to a single executable script. I have not been able to get your branch to run tests on JRuby yet.

@janko
Copy link
Author

janko commented Apr 17, 2015

Yeah, the only reason why I linked you the PR was because I wasn't able to compose a single executable script which fails (I tried to set all encodings as those in the tests, but it didn't work, the script passed). I will try some more.

@scarfacedeb
Copy link

@janko-m @headius I was able to reproduce the issue using your example in jruby-9.0.0.0.rc1:

require "forwardable"

# This is the crucial part
Encoding.default_internal = Encoding::UTF_8

IoWrapper = Struct.new(:io) do
  extend Forwardable
  delegate [:read, :eof?, :close, :size] => :io
end

io = File.open('spec/refile/fixtures/image.jpg', encoding: "ascii-8bit") # == 'rb' mode
# Now io.external_encoding is equal to ASCII-8BIT, just like in the tests.
io_wrapper = IoWrapper.new(io)

IO.copy_stream(io_wrapper, "foo.jpg")

When I set Encoding.default_internal = nil, the problem disappears and it works as expected.

In a similar way, I was getting the same error when I tried to save a binary file with refile.
Applying the same fix with Encoding resolved that issue as well.


The original code example was working with jruby, because Encoding.default_internal == nil, but when I tried to require 'rails/all', the error reappeared.

By trial and error, I've found out that rails was setting Encoding.default_internal in rails/railties/lib/rails.rb which caused that error.

scarfacedeb pushed a commit to scarfacedeb/refile that referenced this issue Jun 19, 2015
IO.copy_stream raises Encoding::UndefinedConversionError when it tries
to copy binary stream in jruby 9.0.0.0.rc1.

Related issue: jruby/jruby#2779
@enebo enebo modified the milestone: JRuby 9.0.0.0 Jul 14, 2015
@headius
Copy link
Member

headius commented Nov 9, 2017

I'm going to close this in hopes that it has been fixed by the thousands of commits since 2015.

@headius headius closed this as completed Nov 9, 2017
@headius headius added this to the Invalid or Duplicate milestone Nov 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants