Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jruby + Puma + Zero downtime deployment #4131

Closed
tridevgurung opened this issue Sep 2, 2016 · 11 comments
Closed

Jruby + Puma + Zero downtime deployment #4131

tridevgurung opened this issue Sep 2, 2016 · 11 comments

Comments

@tridevgurung
Copy link

tridevgurung commented Sep 2, 2016

Environment - Any stage

  • JRuby version 9.1.2.0 engine: 2.3.0
  • JRUBY_OPTS -J-Xcompile.invokedynamic=true -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 -J-noverify -Xcompile.mode=OFF
  • Ubuntu 14.04.4 LTS
  • uname -a Linux my-staging-app1 3.13.0-87-generic #133-Ubuntu SMP Tue May 24 18:32:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Grape (0.17.0)

My puma.rb looks like this

pidfile "/home/deploy/app/shared/tmp/pids/puma.pid"
state_path "/home/deploy/app/shared/tmp/state/puma.state"
stdout_redirect "log/stderr.log", "log/stderr.log", true
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'
threads     (ENV["MIN_PUMA_THREADS"] || 2), (ENV["MAX_PUMA_THREADS"] || 8)

Expected Behavior

  • During deployment I want Jruby to be gracefully shutdown so I won't lose any traffic.

Actual Behavior

  • Jruby doesn't gracefully restarts.
  • I tried using "phased-restart" via pumactl -S puma.state phased-restart which sends kill -s USR2 puma.pid

Due to this behaviour, I am left with this option:

Deploy new version, start up puma on socket Y, wait for puma to become
available Move proxy to socket Y Reload proxy; Old connections might still have go through socket X. (Depending on the proxy/config it might kill them...) Kill old puma process (and let it wait till it handled all connections with a given timeout).

@kares
Copy link
Member

kares commented Sep 2, 2016

any details on the actual behaviour (anything being logged/printed) ? do not know about puma details on how it does "zero down-time" but I assume its not JRuby specific. with JRuby based webservers it was possible by starting another Ruby runtime in the same JVM process and letting it take over ... which worked as long as it did not leak.

@tridevgurung
Copy link
Author

https://github.com/puma/puma#platform-constraints

Puma explicitly tells here

Platform Constraints

Because of various platforms not being able to implement certain things, the following differences occur when Puma is used on different platforms:

JRuby, Windows: server sockets are not seamless on restart, they must be closed and reopened. These platforms have no way to pass descriptors into a new process that is exposed to ruby
JRuby, Windows: cluster mode is not supported due to a lack of fork(2)

Now I'm getting more confused whom I should be seeking help here.

@headius https://twitter.com/headius/status/771383040333455361 😰

@kares
Copy link
Member

kares commented Sep 2, 2016

sorry for confusing you - just trying to gather more info if possible on the issue. thought I get details from you guys on how puma is doing the "phased-restart" (as your "left" option sounded reasonably good to me). you're on Linux platform constraints should not apply here. I am sure @headius will step in as well.

@TheKidCoder
Copy link

Puma does phased restarts using workers. Currently, it is not supported on jRuby because of the use of fork(2) is limited (I think) on jRuby (this may have changed with jruby 9.x?).

@kares
Copy link
Member

kares commented Sep 10, 2016

@TheKidCoder thanks ... that is about what I expected. fork won't ever really work on a JVM.
there are JRuby specific alternatives to using zero down-time ... Trinidad got it working at some point and I am sure TorqueBox as well.

@kares kares closed this as completed Sep 10, 2016
@kares kares added this to the Won't Fix milestone Sep 10, 2016
@headius
Copy link
Member

headius commented Sep 10, 2016

@kares Sorry, I should have jumped on this since I kinda prompted it.

I think we may have some options for restarting an app in JRuby without forking. The idea I have would be to make it possible to take the listening sockets and just swap them into a new runtime. The new runtime would load in the background (big startup time mitigation here), then we flip a switch and it starts servicing requests. It would simply be a matter of passing the old runtime's listening channels into the new one.

I'll reopen this and we can do some experimentation.

@TheKidCoder There is not and probably never will be a way to fork JRuby. But I think we can do something almost as good: just start up another JRuby in the same process and pass along the sockets to it.

@TheKidCoder
Copy link

I'd love to contribute anyway I can. This has been a pain point for me for over a year now when using Puma.

@kares
Copy link
Member

kares commented Sep 10, 2016

well, Puma is not using any JRuby runtime APIs ... so I wonder how you'll do it on JRuby's end not being a mess. this would need redoing Puma with JRuby specific code ... anyway I am looking forward to anything.

@headius
Copy link
Member

headius commented Sep 11, 2016

@kares I would do it in Puma mostly.

@headius
Copy link
Member

headius commented Sep 11, 2016

...or as a plugin for Puma that people could install.

@headius
Copy link
Member

headius commented Jul 9, 2020

On second thought, I think this should be driven by a Puma issue. If there are specific things that JRuby can add to support it, that's fine, but this is a very old and amorphous bug. Closing again, as invalid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants