IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

ferrous26 · 2012-11-27T00:22:07Z

Calling IO.select on SSLSocket's is not behaving correctly. In some cases IO.select will never return (even though a timeout has been given) and in other cases it will erroneously report that the socket is ready for reading when it is not actually ready for reading.

I am experiencing this issue using OpenJDK 1.7 on CentOS. The example code below works fine on OS X 10.8.2. In both cases I am using JRuby 1.7.1 (master).

require 'openssl'
require 'socket'

server_ctx             = OpenSSL::SSL::SSLContext.new
server_ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
server_ctx.cert        = OpenSSL::X509::Certificate.new File.read "config/ssl/cert_test.pem"
server_ctx.key         = OpenSSL::PKey::RSA.new File.read "config/ssl/keys_test.pem"

server        = TCPServer.new 0
ssl_server    = OpenSSL::SSL::SSLServer.new server, server_ctx
client        = TCPSocket.new 'localhost', server.addr[1]
ssl_client    = OpenSSL::SSL::SSLSocket.new client, OpenSSL::SSL::SSLContext.new

Thread.new do ssl_client.connect end
server_socket = ssl_server.accept

result = IO.select [ssl_client], nil, nil, 1
if result
  puts ssl_client.read_nonblock 10
else
  p 'OK'
end

Sorry the the example code can't be smaller, but setting up the SSL connection is a bit of a PITA.

The text was updated successfully, but these errors were encountered:

ferrous26 · 2012-11-27T18:37:02Z

This seems similar to #359, except that TCP sockets are working fine for me, I only have the problem with SSLSockets.

ferrous26 · 2012-11-27T20:40:39Z

I've now tried this out in Ubuntu, with the stock openjdk-7-jdk package and JRuby-head. The test script passes most of the time, but occasionally spins at 100% CPU and does not finish.

BanzaiMan · 2012-12-01T06:09:44Z

Could you show us the PEM file you are using?

ferrous26 · 2012-12-01T06:53:47Z

Certificate:

Keys:

BanzaiMan · 2012-12-01T17:35:52Z

Do you have a rough estimate on how often this blocks? Can you try jruby-openssl 0.8.0.pre3?

I've tested it with the current master, Java 7u9 on Mac OS X 10.8.2, and got the following results:

With jruby-openssl 0.7.7:

30
Sat Dec  1 12:30:28 EST 2012
"OK"
jruby gh-414.rb  6.05s user 0.20s system 151% cpu 4.122 total
31
Sat Dec  1 12:30:32 EST 2012
^Cjruby gh-414.rb  50.88s user 17.94s system 105% cpu 1:04.93 total

With jruby-openssl 0.8.0.pre3, I was able to execute 60+ iteration before I stopped.

59
Sat Dec  1 12:23:54 EST 2012
"OK"
jruby gh-414.rb  0.06s user 0.07s system 2% cpu 4.325 total
60
Sat Dec  1 12:23:58 EST 2012
"OK"
jruby gh-414.rb  0.08s user 0.08s system 3% cpu 4.330 total
61
Sat Dec  1 12:24:03 EST 2012
"OK"
jruby gh-414.rb  0.06s user 0.07s system 3% cpu 4.289 total

ferrous26 · 2012-12-03T15:55:17Z

I actually never ran into this issue on OS X, even with jruby-openssl 0.7.x

My issue has been on Linux, I've tried CentOS and Ubuntu.

After upgrading to jruby-openssl 0.8.0.pre3 my tests will now sometimes finish, but I am still seeing the issue ~50% of the time where the tests do not finish and the process spins at 100% CPU.

bporterfield · 2013-01-28T21:16:04Z

I've duplicated this problem, and I think it's actually the reason my web server occasionally hangs for my clients (we sometimes need to read_nonblock for a specific case, and I think that occasionally this is hanging). I also can not duplicate on mac OS X.

My testing platform:

Linux ip-10-29-159-172 3.2.0-27-virtual #43-Ubuntu SMP Fri Jul 6 15:26:20 UTC 2012 i686 i686 i386 GNU/Linux

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5) (6b24-1.11.5-0ubuntu1~12.04.1)
OpenJDK Client VM (build 20.0-b12, mixed mode, sharing)

jruby 1.7.1 (1.9.3p327) 2012-12-03 30a153b on OpenJDK Client VM 1.6.0_24-b24 [linux-i386]

Even when the script works, the CPU still goes through the roof. This is a major problem for us - our clients are losing their ability to use the web server - we would love to see some attention to it!

bporterfield · 2013-01-31T18:45:45Z

I stand corrected - I've also been able to reproduce this issue on a mac. Here is the platform:

Mac OS X 10.7.5

jruby 1.7.1 (1.9.3p327) 2012-12-03 30a153b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_04-b21 [darwin-x86_64]

java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)

headius · 2016-04-05T06:45:51Z

IO.select(ssl_socket) is not reliable on any implementation, due to the buffering that occurs internally for most SSL socket wrappers. Basically, IO.select does not know about those buffers. This is a known issue and we and ruby-core have not found a satisfactory solution.

See https://bugs.ruby-lang.org/issues/8875.

Now that may or may not explain this issue, but it would be very easy to get stuck in a loop if select kept claiming data was available (because the socket has data) but the SSLSocket does not consume it properly (because it's missing some handshake or other sequence).

I'm going to close this, since it's both very old and relates to a known issue with IO.select(ssl_socket) that neither we nor MRI plan to fix.

ostinelli · 2016-04-27T08:38:03Z

I'm going to close this, since it's both very old and relates to a known issue with IO.select(ssl_socket) that neither we nor MRI plan to fix.

Indeed, however SSLSocket in the MRI has the method pending implemented, which can then allow to solve this issue by first reading the contents of the SSL buffer before IO.select.

However in JRuby this method is still not implemented. Because of this, I cannot add JRuby support for a library that uses SSL Sockets. Are there any plans to implement pending?

BTW thank you @headius for the great work :)

headius · 2016-05-02T17:27:49Z

@ostinelli I had no idea the pending method existed! Could you file a jruby-openssl issue about it, ideally including information from MRI (feature request, commit number, etc)? We can get it added in the next round!

headius · 2016-05-02T17:28:17Z

@kares @mkristian It should be reasonably easy to implement pending for SSLSocket, eh?

kares · 2016-05-02T17:43:07Z

@headius not sure about the SSL engine behaviour - if wrap always returns all data than the non-empties buffer might be holding the pending bytes...

ferrous26 mentioned this issue Dec 3, 2012

IO.select breaks SSLSocket#read_nonblock on Linux #423

Closed

headius closed this as completed Apr 5, 2016

headius added this to the Won't Fix milestone Apr 5, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Sponsors

IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

ferrous26 commented Nov 27, 2012

ferrous26 commented Nov 27, 2012

ferrous26 commented Nov 27, 2012

BanzaiMan commented Dec 1, 2012

ferrous26 commented Dec 1, 2012

BanzaiMan commented Dec 1, 2012

ferrous26 commented Dec 3, 2012

bporterfield commented Jan 28, 2013

bporterfield commented Jan 31, 2013

headius commented Apr 5, 2016

ostinelli commented Apr 27, 2016

headius commented May 2, 2016

headius commented May 2, 2016

kares commented May 2, 2016

IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

Comments

ferrous26 commented Nov 27, 2012

ferrous26 commented Nov 27, 2012

ferrous26 commented Nov 27, 2012

BanzaiMan commented Dec 1, 2012

ferrous26 commented Dec 1, 2012

BanzaiMan commented Dec 1, 2012

ferrous26 commented Dec 3, 2012

bporterfield commented Jan 28, 2013

bporterfield commented Jan 31, 2013

headius commented Apr 5, 2016

ostinelli commented Apr 27, 2016

headius commented May 2, 2016

headius commented May 2, 2016

kares commented May 2, 2016