Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

Closed
ferrous26 opened this issue Nov 27, 2012 · 13 comments
Closed

IO.select problems with SSL sockets on JRuby 1.7 on CentOS #414

ferrous26 opened this issue Nov 27, 2012 · 13 comments
Labels
Milestone

Comments

@ferrous26
Copy link
Contributor

Calling IO.select on SSLSocket's is not behaving correctly. In some cases IO.select will never return (even though a timeout has been given) and in other cases it will erroneously report that the socket is ready for reading when it is not actually ready for reading.

I am experiencing this issue using OpenJDK 1.7 on CentOS. The example code below works fine on OS X 10.8.2. In both cases I am using JRuby 1.7.1 (master).

require 'openssl'
require 'socket'

server_ctx             = OpenSSL::SSL::SSLContext.new
server_ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
server_ctx.cert        = OpenSSL::X509::Certificate.new File.read "config/ssl/cert_test.pem"
server_ctx.key         = OpenSSL::PKey::RSA.new File.read "config/ssl/keys_test.pem"

server        = TCPServer.new 0
ssl_server    = OpenSSL::SSL::SSLServer.new server, server_ctx
client        = TCPSocket.new 'localhost', server.addr[1]
ssl_client    = OpenSSL::SSL::SSLSocket.new client, OpenSSL::SSL::SSLContext.new

Thread.new do ssl_client.connect end
server_socket = ssl_server.accept

result = IO.select [ssl_client], nil, nil, 1
if result
  puts ssl_client.read_nonblock 10
else
  p 'OK'
end

Sorry the the example code can't be smaller, but setting up the SSL connection is a bit of a PITA.

@ferrous26
Copy link
Contributor Author

This seems similar to #359, except that TCP sockets are working fine for me, I only have the problem with SSLSockets.

@ferrous26
Copy link
Contributor Author

I've now tried this out in Ubuntu, with the stock openjdk-7-jdk package and JRuby-head. The test script passes most of the time, but occasionally spins at 100% CPU and does not finish.

@BanzaiMan
Copy link
Member

Could you show us the PEM file you are using?

@ferrous26
Copy link
Contributor Author

Certificate:

-----BEGIN CERTIFICATE-----
MIIDkTCCAzugAwIBAgIJANdvjRGkTITaMA0GCSqGSIb3DQEBBQUAMIG5MQswCQYD
VQQGEwJDQTEQMA4GA1UECBMHT250YXJpbzEQMA4GA1UEBxMHTWFya2hhbTEaMBgG
A1UEChMRTWFya2V0Y2lyY2xlIEluYy4xFDASBgNVBAsTC0VuZ2luZXJkaW5nMS0w
KwYDVQQDFCRhdHRhY2htZW50c2VydmVyLm1hcmtldGNpcmNsZS5jCHRlc3QxJTAj
BgkqhkiG9w0BCQEWFm1yYWRhQG1hcmtldGNpcmNsZS5jb20wHhcNMTIwOTE4MTQ1
NTA3WhcNMTUwOTE4MTQ1NTA3WjCBuTELMAkGA1UEBhMCQ0ExEDAOBgNVBAgTB09u
dGFyaW8xEDAOBgNVBAcTB01hcmtoYW0xGjAYBgNVBAoTEU1hcmtldGNpcmNsZSBJ
bmMuMRQwEgYDVQQLEwtFbmdpbmVyZGluZzEtMCsGA1UEAxQkYXR0YWNobWVudHNl
cnZlci5tYXJrZXRjaXJjbGUuYwh0ZXN0MSUwIwYJKoZIhvcNAQkBFhZtcmFkYUBt
YXJrZXRjaXJjbGUuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALXXlUSlpfCy
2o/EHJAcWxJjBVq9zgQoqwtq3eXKSuQRYel82WOp2NyerH7IdxODHyvN10zeVBGD
i9KqEFNaaZcCAwEAAaOCASIwggEeMB0GA1UdDgQWBBT6enqWrgWwTVDGLR4H3eLR
SRQ3kzCB7gYDVR0jBIHmMIHjgBT6enqWrgWwTVDGLR4H3eLRSRQ3k6GBv6SBvDCB
uTELMAkGA1UEBhMCQ0ExEDAOBgNVBAgTB09udGFyaW8xEDAOBgNVBAcTB01hcmto
YW0xGjAYBgNVBAoTEU1hcmtldGNpcmNsZSBJbmMuMRQwEgYDVQQLEwtFbmdpbmVy
ZGluZzEtMCsGA1UEAxQkYXR0YWNobWVudHNlcnZlci5tYXJrZXRjaXJjbGUuYwh0
ZXN0MSUwIwYJKoZIhvcNAQkBFhZtcmFkYUBtYXJrZXRjaXJjbGUuY29tggkA12+N
EaRMhNowDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAANBAARxU8fvpEEoUEIW
vEvar8P7LwjgEJonT1K3AGF3PPgW5vCDhBomPre/xQfdzcae1Zukwim1KJuVTK/a
OaKgrgg=
-----END CERTIFICATE-----

Keys:

-----BEGIN RSA PRIVATE KEY-----
MIIBOQIBAAJBALXXlUSlpfCy2o/EHJAcWxJjBVq9zgQoqwtq3eXKSuQRYel82WOp
2NyerH7IdxODHyvN10zeVBGDi9KqEFNaaZcCAwEAAQJAE5PybkGnbBLVlUT4Dkv/
QiuHGA33cFswjzoiSetNxOArw2LC2bqfOYzKpNdBTOGg9FSQrBKSkBmPXeT/Ym2a
AQIhANsvIduBxIT3Z6xsDVCvYfVXLbkqUo0w6JbVkR+MUmJxAiEA1GLA7uqx60EU
BKhuTSJtSYqzrFWDHm+Mdr/8mEWwgIcCIGertWHz0RXrR1ggi3O8YKMApxnL0NwD
ZB/jyA6Z+mWhAiB+zv/XJvZYOqETLuW81GNgV5pzuRLypzO862CMIoACHQIgPBET
NhF3prgLK3ifQyiY4exyfEcS4EFlunYTtjvHWjk=
-----END RSA PRIVATE KEY-----

@BanzaiMan
Copy link
Member

Do you have a rough estimate on how often this blocks? Can you try jruby-openssl 0.8.0.pre3?

I've tested it with the current master, Java 7u9 on Mac OS X 10.8.2, and got the following results:

With jruby-openssl 0.7.7:

30
Sat Dec  1 12:30:28 EST 2012
"OK"
jruby gh-414.rb  6.05s user 0.20s system 151% cpu 4.122 total
31
Sat Dec  1 12:30:32 EST 2012
^Cjruby gh-414.rb  50.88s user 17.94s system 105% cpu 1:04.93 total

With jruby-openssl 0.8.0.pre3, I was able to execute 60+ iteration before I stopped.

59
Sat Dec  1 12:23:54 EST 2012
"OK"
jruby gh-414.rb  0.06s user 0.07s system 2% cpu 4.325 total
60
Sat Dec  1 12:23:58 EST 2012
"OK"
jruby gh-414.rb  0.08s user 0.08s system 3% cpu 4.330 total
61
Sat Dec  1 12:24:03 EST 2012
"OK"
jruby gh-414.rb  0.06s user 0.07s system 3% cpu 4.289 total

@ferrous26
Copy link
Contributor Author

I actually never ran into this issue on OS X, even with jruby-openssl 0.7.x

My issue has been on Linux, I've tried CentOS and Ubuntu.

After upgrading to jruby-openssl 0.8.0.pre3 my tests will now sometimes finish, but I am still seeing the issue ~50% of the time where the tests do not finish and the process spins at 100% CPU.

@bporterfield
Copy link

I've duplicated this problem, and I think it's actually the reason my web server occasionally hangs for my clients (we sometimes need to read_nonblock for a specific case, and I think that occasionally this is hanging). I also can not duplicate on mac OS X.

My testing platform:

Linux ip-10-29-159-172 3.2.0-27-virtual #43-Ubuntu SMP Fri Jul 6 15:26:20 UTC 2012 i686 i686 i386 GNU/Linux

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5) (6b24-1.11.5-0ubuntu1~12.04.1)
OpenJDK Client VM (build 20.0-b12, mixed mode, sharing)

jruby 1.7.1 (1.9.3p327) 2012-12-03 30a153b on OpenJDK Client VM 1.6.0_24-b24 [linux-i386]

Even when the script works, the CPU still goes through the roof. This is a major problem for us - our clients are losing their ability to use the web server - we would love to see some attention to it!

@bporterfield
Copy link

I stand corrected - I've also been able to reproduce this issue on a mac. Here is the platform:

Mac OS X 10.7.5

jruby 1.7.1 (1.9.3p327) 2012-12-03 30a153b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_04-b21 [darwin-x86_64]

java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)

@headius
Copy link
Member

headius commented Apr 5, 2016

IO.select(ssl_socket) is not reliable on any implementation, due to the buffering that occurs internally for most SSL socket wrappers. Basically, IO.select does not know about those buffers. This is a known issue and we and ruby-core have not found a satisfactory solution.

See https://bugs.ruby-lang.org/issues/8875.

Now that may or may not explain this issue, but it would be very easy to get stuck in a loop if select kept claiming data was available (because the socket has data) but the SSLSocket does not consume it properly (because it's missing some handshake or other sequence).

I'm going to close this, since it's both very old and relates to a known issue with IO.select(ssl_socket) that neither we nor MRI plan to fix.

@headius headius closed this as completed Apr 5, 2016
@headius headius added this to the Won't Fix milestone Apr 5, 2016
@ostinelli
Copy link

I'm going to close this, since it's both very old and relates to a known issue with IO.select(ssl_socket) that neither we nor MRI plan to fix.

Indeed, however SSLSocket in the MRI has the method pending implemented, which can then allow to solve this issue by first reading the contents of the SSL buffer before IO.select.

However in JRuby this method is still not implemented. Because of this, I cannot add JRuby support for a library that uses SSL Sockets. Are there any plans to implement pending?

BTW thank you @headius for the great work :)

@headius
Copy link
Member

headius commented May 2, 2016

@ostinelli I had no idea the pending method existed! Could you file a jruby-openssl issue about it, ideally including information from MRI (feature request, commit number, etc)? We can get it added in the next round!

@headius
Copy link
Member

headius commented May 2, 2016

@kares @mkristian It should be reasonably easy to implement pending for SSLSocket, eh?

@kares
Copy link
Member

kares commented May 2, 2016

@headius not sure about the SSL engine behaviour - if wrap always returns all data than the non-empties buffer might be holding the pending bytes...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants