-
-
Notifications
You must be signed in to change notification settings - Fork 925
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Securerandom and KVM virtual machines #1896
Comments
What version of JRuby are you running? 1.7.11 still starts by seeding from /dev/random, but massively reduces the number of times this is necessary. More details: http://blog.mogotest.com/2014/03/11/faster-securerandom-in-jruby-1.7.11/ |
I've been using 1.7.12 I think the problem is only apparent when /dev/random is depleted, it takes quite some time for available randomness to get built up so if it runs out you can see the 2 calls to SecureRandom.hex took over 3 minutes each... /dev/random really is that slow to provide data on KVM VMs |
In that case, I think your only real option is to override the seed source as you've found. While seeding from /dev/urandom is an option (and apparently is a fallback in MRI), I think it was determined to be too risky to make such a sweeping change in a point release. There was also some disagreement as how to interpret the Linux man page warning about /dev/urandom not being secure. @headius can probably provide a bit more insight there. In the meanwhile, if the risk is appropriate for you, you do have a valid solution in front of you. |
I'm okay with using this workaround. If, like you said, there was a deliberate decision to avoid changing to urandom right now, then it may be worth mentioning it in the docs, to someone who's at a loss as to why their app takes 5 minutes to start or certain actions that they think are lightweight take an equally large time, I mean it cost me a few days trying to work out what I was doing wrong. Hosting on Linode and other KVM hosts has got to be fairly popular these days so I would say it's a common use case for JRuby. |
Ultimately, it's doing what Java would do. Arguably the host should ensure a source of entropy given how many applications there do use /dev/random. Having said that, adding it to the wiki sounds reasonable. (And FWIW, I'm not on the dev team . . . just a guy that spent a lot of time trying to speed up Rails in JRuby). |
Put another way, if /dev/random is causing you to block, someone else on that machine must be aggressively reading it. Sounds like a good DoS vector. I'd just think the host would have a way to guard against this, like with CPU steal. |
You almost had me for a minute there! ;) KVM doesn't distribute the host's /dev/random data amongst the Guest VMs, they have to make their own (which is a lot harder when you've not got physical devices with the associated noise). I validated this with my own KVM host, shut down all the other guests and there was virtually no data coming through /dev/random, but logging in to the host, there was plenty. |
@nirvdrum , a lot of headless virtual servers don't have keyboards or mice and only rarely get network packets, which often leads to a depleted entropy pool even under light use. There seems to be some controversy over whether /dev/urandom is an acceptable substitute for /dev/random in such cases. If /dev/urandom is not an option for whatever reason, there's a daemon called haveged that adds entropy to the pool by re-running the same code many times and drawing entropy from the minute variations in execution time. It can't hurt to have more entropy, in any case. |
I never noticed this bug, but we had another user discover that an entropy server helped their startup time. This is fascinating! I will be adding an entry to the wiki page on startup time. @trinode how much did your times improve once you set up an entropy server? |
@headius I managed to make do with using /dev/urandom, I never used an entropy server, but it shaved that 6 minute delay from the startup. I've had to go back to MRI for now (needing things like activerecord-postgis-adapter and other things which seem to not support JRuby in a timely fashion. |
I've experienced this many times with Under Debian-like Linux flavors like Mint and Ubuntu, this gets that done:
|
JRuby prefers the non-blocking variant when available + SecureRandom has been updated to re-use the same path for generating random values (in 9.1.11/12) ... closing |
I've been using jRuby in development for a while now and now I've set up a virtual machine host here as the staging server, and a production server is hosted at Linode (I just mention that as it affects that server too and they know what they're doing...)
here's a profile I did trying to work out how come various things were taking forever, including starting my tests and actually starting the site in puma.
Over 3 minutes per call to securerandom (it turns out on Virtual machines - my CI server, my staging server (self hosted) and my production box (Linode hosted) that entropic data is low in terms of volume, you can cat /dev/random and it runs out of data pretty fast, /dev/urandom is not affected)
So I did some research and found I could force the JVM to use urandom
( -J-Djava.security.egd=/dev/urandom) the timing of securerandom becomes perfectly good.
one thing I noticed though, if it was left for a while the /dev/random buffer had enough entropy backed up to be able to satisfy securerandom immediately for the first few calls.
I use securerandom a fair amount for generating UUIDs so it's really crippling my app, I have the option workaround to use urandom for now, but I bet a lot of people won't dig that far into it, assuming their app (that can take up to 8 minutes to start in my case) is simply crashing on jRuby or that the VM they use hasn't got the horsepower for jRuby.
The text was updated successfully, but these errors were encountered: