Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lookup.loadCertificateOrCRLFile consumes more memory because of frequent memory allocation at Certificate.matches #86

Closed
frsyuki opened this issue Mar 15, 2016 · 7 comments

Comments

@frsyuki
Copy link

frsyuki commented Mar 15, 2016

Overview

  • Since jruby-openssl 0.9.8, Lookup.loadCertificateOrCRLFile calls Store.matchedObject, and it calls Certificate.matches method for each stored certificates.
  • Certificate.matches method calls x509.getSubjectX500Principal() method. This method internally encodes aux certificate using Bouncy Castle's ASN1OutputStream.
  • This encoding procedure creates many objects.
  • It impacts GC performance badly.

Background

We upgraded JRuby 9.0.0.0 to 9.0.5.0 and found that our application started consuming much more memory. This application uses httpclient.gem to fetch data from HTTPS server. Here is its code:
https://github.com/treasure-data/embulk-input-sfdc/blob/master/lib/embulk/input/sfdc_api/api.rb

Here is a screenshot of memory profiler:

  • JRuby 9.0.0.0 (which uses jruby-openssl v0.9.7)
  • JRuby 9.0.5.0 (which uses jruby-openssl v0.9.11)

This shows that 9.0.5.0 allocates objects more frequently and triggers GC frequently (2x - 3x more frequent).

Memory profiler shows that under Lookup.loadCertificateOrCRLFile method, JRuby 9.0.0.0 doesn't allocate many objects but 9.0.5.0 allocates many objects at Store.addCertificate in addition to regular allocation at PEMInputOutput.readPEM:

  • JRuby 9.0.0.0
  • JRuby 9.0.5.0

Under Store.addCertificate, following stack is allocating majority of objects:

  • Store.matchedObject
    • Certificate.matches
      • X509AuxCertificate.getSubjectX500Principal
        • ASN1OutputStream.writeObject
          • DERSequence.encode
            • ...

I think that Store.matchedObject method is just a lookup method but X509AuxCertificate.getSubjectX500Principal is allocating many objects unexpectedly.

Expected behavior

I think that Store.matchedObject or Certificate.matches can cache the result of X509AuxCertificate.getSubjectX500Principal() to reduce object allocation.

Workaround

Memory allocation becomes much less frequent if I add jruby.openssl.x509.lookup.cache=8 system property:

  • JRuby 9.0.5.0 + jruby.openssl.x509.lookup.cache=8

But this option is disabled by default now.

@kares
Copy link
Member

kares commented Mar 16, 2016

believe this is right, added jruby.openssl.x509.lookup.cache property as a HotSpot security provider contention work-around. its good that you're able to use it but I'm not sure caching by default is a good idea for a security library (although in this case it might be harmless).

any ideas on reducing Certificate.matches without caching? // cc @mkristian

@mkristian
Copy link
Member

@kares I was thinking about this. bit hard without caching. as Certificate.matches could "cache" the hashCode of x509.getSubjectX500Principal() and first compares the hashCodes before diving into the X500Prinicipal object:
https://github.com/jruby/jruby-openssl/blob/master/src/main/java/org/jruby/ext/openssl/x509store/Certificate.java#L55

well, just an idea and on a first glance I hope it sound reasonable ;)

@kares
Copy link
Member

kares commented Mar 17, 2016

@mkristian good point, although we might need to look into how hashCode is implemented in BC ...
so that we do not end up doing the same encoding while doing it.

kares added a commit that referenced this issue Apr 20, 2016
@kares
Copy link
Member

kares commented Apr 21, 2016

@frsyuki if possible could you guys try a --pre release from https://oss.sonatype.org/content/repositories/snapshots/rubygems/jruby-openssl/0.9.17.dev-SNAPSHOT/ ... did an experiment to only use BC certificate's getSubjectX500Principal when comparing against a non-BC cert. wonder what memory consumption would behave in cases such as yours (with lookup caching turned off).

@kares
Copy link
Member

kares commented Jun 1, 2016

seems like its caused by extensive memory use with BC 1.54 where they need the provider to be registered.
details at #94 ... work-around would be to port over some of the X.509 factory code so it does not need to instantiate a new provider with every certificate. should get "huge" memory consumption drop dramatically.

@headius
Copy link
Member

headius commented Jul 11, 2016

This is fixed, isn't it?

@kares
Copy link
Member

kares commented Jul 12, 2016

@headius yep it should be, thanks

@kares kares closed this as completed Jul 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants