-
-
Notifications
You must be signed in to change notification settings - Fork 925
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Attempt to get output from PTY.spawn periodically hangs #3020
Comments
Trying When the process hangs on Process.wait(the_pid, Process::WNOHANG) However, what causes the spawn to sometimes fail is not known. I also had to break apart the arguments on the spawn otherwise it would not work: PTY.spawn('/bin/echo', 'blah') do ... After much trial and error I was able to get it to work on JRuby with the following code that includes an automatic retry for when the spawn sometimes fails: Note that calling |
The subprocess support (spawn, popen, etc) on JRuby 1.7 is fairly limited, since it still uses JDK libraries for all process launching. Those libraries are known to have many problems, especially as relates to interactive IO with the subprocess. I doubt we'll be able to do much better there. However I'd expect 9k to be better here, and it sounds like it's almost there. @reidmorrison I guess you found this bug report looking for info about your 9k issue? For now we can keep using this bug, but I suspect the reasons for 1.7 failing are different from 9k. If you can provide a script I can run that reproduces the intermittent spawn failure, I can have a look. I don't know why it would work sometimes and not others. For reference, JRuby 9k uses |
This standalone script reproduces the intermittent failed spawn with JRuby 9.1.7.0 on Mac OS X 10.12.3 require 'pty'
require 'expect'
itr = 0
while true
itr = itr + 1
PTY.spawn('/bin/echo', 'blah') do |reader, writer, pid|
begin
if reader.expect(/blah.*/, 5)
puts "#{itr} - #{pid} - Successful"
else
puts "#{itr} - #{pid} - !!!! FAILED to spawn !!!!!"
end
ensure
reader.close
writer.close
Process.wait(pid, Process::WNOHANG)
end
end
end I also tried this script with similar results with JRuby 9.1.5.0 on Linux: |
Reproduced! |
Wow...ok, this is a problem. Our pty is currently implemented using |
For those of you using |
Ok, I spent a few minutes poking around pty.c from MRI. It doesn't look great. In general it appears to use most of the same internals as Kernel.spawn, which we've largely ported over to Java for JRuby. However it does NOT appear to support running on platforms that don't support So this exact implementation isn't going to work for us. Now, a bit of good news. So far I've seen nothing that we shouldn't be able to do with I still believe it should be possible to implement the library, in pure-Ruby, using just Kernel.spawn. |
Another option for us would be to finally ship a small native library that can do forkpty and related calls all in one JNI downcall. This is how the JVM itself launches subprocesses, via a JNI call that eventually does both fork and exec in rapid succession. Because the entire JNI call is JVM safepoint-aware, it should avoid issues with JVM threads and signals firing inbetween the fork and exec. I'm still trying to wrap my head around what forkpty actually does, though, to see if we can't emulate it with a simple spawn. |
For those following along...this isn't likely to be fixed any time soon, due to the discovery of fork. If you are able to use something known to work, like Kernel.spawn, you're going to be in better shape. |
@reidmorrison @camlow325 Can you guys tell us why you need a PTY here? The only examples given so far are |
@headius Thanks for looking into this. TBH, it's been long enough now that I don't remember the exact details for our original use case. I believe one of my colleagues was wanting to programmatically invoke a command line tool which prompts for a passphrase on stdin. He wanted to be able to feed data into stdin and capture the output that the tool produced on stdout and thought to use |
@camlow325 Ok thanks. Yeah anything that's using e.g. stty or that requires terminal characteristics like window size would need a pty, but simple subcommands should work fine with a plain spawn. |
The reason I am using the Using Unfortunately the sftp ruby gem is extremely slow and does not support uploading files over 2gb, so using I only submitted the above script to reproduce the issue, to help out where I can. But I totally agree that in every other case it is a complete overkill and I prefer to use one of the Adding a simple retry for now in this very rare case of requiring a new pty to workaround a Compared to other things like performance optimizations for keyword arguments, this would be very low on the priority list, almost a nice-to-have. |
I have good news!!! I discovered we have another forgotten bug, #3185, in which @Freaky provides a rewritten pty.rb that does not use fork! I'm testing it locally. Unfortunately there's a small bug in JRuby that prevents it working properly on 9k, so there still needs to be a patch. Specifically, when opening an IO from a TTY file descriptor, it mistakenly thinks the descriptor is closed and errors out. However, with that patched...
It's all good! I'm going to push @Freaky's pty.rb and let you give it a shot, @reidmorrison. He claims it even passes MRI tests, so we'll just go for it. |
Excellent news, Thank you 👍 |
We're experiencing periodic hangs during attempts to read the output from or wait on a process created by a call to
PTY.spawn
. Here's a snippet of code which demonstrates the problem:From the command line, we just run this as
jruby ./thefile.rb
. We've observed this code fail in multiple ways:Most often, the process hangs on the
cmd_out.getc.chr
call.Sometimes, the process hangs on the
Process.wait (pid)
call.Much less frequently - the data that the
cmd_out.getc.chr
andcmd_out.gets
calls read back appears to be the path to the jruby jar + an error message. For example:It is pretty uncommon to get through more than about about 150 iterations or so before one of the failures above happens. The third failure, when it occurs, is usually on the first
PTY.spawn
call.I've been able to reproduce this failure on multiple versions of JRuby - including 1.7.19, 1.7.20, 9.0.0.0.pre2. I haven't seen the program run indefinitely without failing on any version of JRuby, so I suspect that this may never have worked.
Is there something obviously wrong about the way we're using
PTY.spawn
here, or isPTY.spawn
basically not functional at this point?The text was updated successfully, but these errors were encountered: