Rails csfr_meta_tag requires minutes to render due to jruby use of /dev/random on a virtual machine #4685
Comments
roboyeti
changed the title
|
I hate to give bad news, but obviously we're not going to be patching anything for a JRuby 1.5.x release. 😄 As you've found this is a known issue that affects not only JRuby but many other random number generators. We continue to wrestle with the crypto libraries on the JVM, trying to find the right balance of RNG and PRNG but there are many combinations. As an example, we recently thought we'd solved this by explicitly using an OpenSSL-like PRNG seeded from /dev/urandom. It stopped our blocking for a while. Then a configuration change in the JVM seemed to start using /dev/random again. Then an update of our Bouncy Castle crypto library started using /dev/random in that library as well. It's rather frustrating. At this point, given the many variables, we have generally been recommending the haveged workaround. Otherwise there's wiki articles about configuring JVM crypto and other flags to do the right thing. But I'm not sure we have a good way to "solve" this other than educating folks. If you are interested in prototyping some mechanism for "probing" the RNG to pick the right configuration, we'd happily work with you on that. I expect the main challenge will be that it's hard or impossible to query all the various layers of JVM security to know what they're currently using, and varying ways to configure them to use something specific. |
|
Well, before you get to excited about the version of jruby this is under, that was a copy and paste mistake from the wrong terminal window, jruby 1.5.6 is the default on the system, but I am actually running the rails app from the following: jruby 9.1.12.0 (2.3.3) 2017-06-15 33c6439 OpenJDK 64-Bit Server VM 24.131-b00 on 1.7.0_131-b00 +jit [linux-x86_64] Sorry for the mistake. I was a bit annoyed and exhausted after being up for hours trying to figure out why it wasn't working to hit an impending deployment deadline. You just have to ask yourselves over there, if rails is the "killer app" of ruby, and right now, with the most basic rails install not only not able to work properly under jruby and the only way to actually find out about the problem is to already kind of know what the problem is in the first place is probably not ideal for converting people into using jruby. A wiki article that requires knowing enough about the issue to find it is probably not the best solution. I understand it is a frustrating problem, especially as the "man in the middle" role of jruby. I will start thinking about possible options and digging into the issue to see if I can help at all. |
|
I believe I have a similar problem going on trying to access a '.js.erb' resource via link_to (remote) in rails 4 and jruby 2.3.3 (linux vmware host). I am getting a random (~1min) delay accessing this resource. I am also using OpenJDK so thinking that might be related. Using 'jstack' I was able to pull up the below stack trace (references IO inside bouncycastle lib): |
|
@roboyeti @ryanrolland Circling back to this for a moment... I am not sure what the best solution is. Apparently most Java folks don't run into this because otherwise there'd be more posts about what to do. One thing I can recommend to all of you is this flag: This tells the JDK to use urandom instead of random, which generally avoids the entropy exhaustion problem on Linux. What To Do?At this point I think we have a few options:
Ping @enebo since this continues to bite people. |
See #4685, jruby/jruby-openssl#135, among others.
|
Ok, I went ahead and did it. Seems like this just needs to be the default given the endless problems we have had. The patch in 1a4f888 can be done to any JRuby version, for those following along. We'll also need to get this into the native jruby-launcher before we can call this good. |
See jruby/jruby#4685 and many others.
|
Ok, I have updated JRuby's bash script and the native launcher gem (gem install jruby-launcher) to both force the entropy source to /dev/urandom. I did not update the Windows executable, but I don't believe this property does anything in Windows anyway. Going to call this fixed, as of JRuby 9.1.15 and jruby-launcher 1.1.3. |
Environment
Expected Behavior
Expected that after conversion of MRI to jruby on rails application for it to render views in a reasonable amount of time (at least as fast as MRI rendering) on a virtual machine.
Actual Behavior
Client requests to even a basic rails view using a layout that includes csrf_meta_tag may require several minutes to render. Welcome#index works fine, as does using "render layout: false" in controller method, but including the line <%= csrf_meta_tags %> will drop response time to minutes per request at best. Presumably, /dev/random is used by jruby and is not getting populated fast enough on a virtual machine.
As common as VM are, how bad blocking on /dev/random is, and how common csrf_meta_tags are used in rails, a better out of the box solution should be provided in jruby. Really, /dev/urandom should probably be used (if not initially, then as a fallback to /dev/random blocking). Most of the "concerns" proposed on the man page of /dev/urandom are extremely unlikely or situational (see links below).
Failing that, perhaps a timeout on /dev/random and throwing a warning to that effect would be great before trying /dev/random again, as the problem can be difficult to isolate and throwing warnings in a read loop with a timeout seems a reasonable way to inform jruby users why their software is just suddenly frozen for unpredictable (or at least not easily predictable) amounts of time, but works fine under MRI.
See:
https://www.2uo.de/myths-about-urandom/
https://unix.stackexchange.com/questions/324209/when-to-use-dev-random-vs-dev-urandom
Used work around.
Installing "haveged" resolved the issue as a work around.
Related issues (closed ... but not very resolved by 9.1.12.0... ): #1896
The text was updated successfully, but these errors were encountered: