I don't know if this is the place to submit this issue but there seems to be a pretty big incompatability between the JRuby and MRI Implementation of OpenSSL regarding RSA keys.

Basically if you generate and export an RSA key pair in JRuby you end up with keys looking like this:

-----BEGIN RSA PUBLIC KEY----- ...somekey... -----END RSA PUBLIC KEY-----

Those keys however are considered invalid when trying to read them under MRI - openssl then throws an Neither PUB key nor PRIV key:: nested asn1 error

Now the keys generated under MRI have different headers:

-----BEGIN PUBLIC KEY----- ...somekey... -----END PUBLIC KEY-----

Once you take the JRuby keys and remove RSA from the header and footer the key gets accepted just fine under MRI.
Here's the code that I use to generate the key pair. The same code results in different output under MRI and JRuby - different headers.

    require 'openssl'
    key = OpenSSL::PKey::RSA.new(2048)

    cipher = OpenSSL::Cipher.new 'AES-128-CBC'
    pass_phrase = 'some_pass_phrase'

    key_secure = key.export cipher, pass_phrase

    open public_key_path, 'w' do |io| io.write key.public_key.to_pem end

    open private_key_path, 'w' do |io| io.write key_secure end

The JRuby generated key is of course read without problems under JRuby. However trying to read that same public key generated with JRuby under MRI results in error.

PUBLIC_KEY = OpenSSL::PKey::RSA.new File.read(public_key_path)

The described behaviour is observed in JRuby 1.7.17 and MRI 2.1.3.

Basically JRuby generates PKCS#1 RSAPublicKey (PEM header: BEGIN RSA PUBLIC KEY) and MRI generates X.509 SubjectPublicKeyInfo (PEM header: BEGIN PUBLIC KEY).
Shouldn't the behaviour be consistent under all platforms?