open3.rb broken in JRuby #5018

cshupp1 · 2018-01-29T15:34:28Z

Consider the following:

irb(main):002:0* JRUBY_VERSION
=> "9.1.15.0"
irb(main):003:0> java.lang.System.getProperties['os.name']
=> "Windows 10"
irb(main):006:0> stout, stderr, status = Open3.capture3({}, "bundle exec rake routes")
IOError: Cannot run program "bundle" (in directory "C:\work\VSO\vso"): CreateProcess error=2, The system cannot find the file specified
        from org/jruby/RubyIO.java:4845:in `popen3'
        from uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jruby/open3_windows.rb:74:in `popen3'
        from uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jruby/open3_windows.rb:272:in `capture3'
        from (irb):6:in `<eval>'
        from org/jruby/RubyKernel.java:994:in `eval'
        from org/jruby/RubyKernel.java:1292:in `loop'
        from org/jruby/RubyKernel.java:1114:in `catch'
        from org/jruby/RubyKernel.java:1114:in `catch'
        from uri:classloader:/META-INF/jruby.home/bin/jirb:13:in `<main>'

Now consider MRI ruby

irb(main):025:0> RUBY_VERSION
=> "2.3.3"
irb(main):029:0>       sttdout, stderr, status = Open3.capture3({}, "bundle exec rake routes")
=> ["\"The version is \"\n\"Setting rails war to use production\"\n    Prefix Verb URI Pattern                      Controller#Action\n      root GET  /                                welcome#index\nfetch_text GET  /fetch_text/
fetch_text(.:format) fetch_text#fetch_text\n", "C:/work/VSO/vso/config/environment.rb:5: warning: already initialized constant WINDOWS\nC:/work/VSO/vso/config/environment.rb:7: warning: already initialized constant CATALINA_HOM
E\n", #<Process::Status: pid 11300 exit 0>]

In MRI ruby we can make a call to bundler. JRuby we cannot.

Please note the following in JRuby:

irb(main):007:0> stout, stderr, status = Open3.capture3({}, "which bundle")
=> ["/c/work/VSO/gem_home/bin/bundle\n", "", #<Process::Status: pid 7304 exit 0>]
irb(main):008:0>

So JRuby should be able to find it.

I have written up the following webpacker bug which shows a workaround, but it just could be that the root cause of the issue is with JRuby.

rails/webpacker#1175

The text was updated successfully, but these errors were encountered:

enebo · 2018-01-29T16:02:29Z

@cshupp1 do you know if this is a regression or likely something which has been broken for the whole 9.1.x series?

cshupp1 · 2018-01-29T16:26:09Z

@enebo It fails for me with 9.1.8 and 9.0.4.0. Can you reproduce it?

enebo · 2018-01-29T16:33:33Z

@cshupp1 we implement capture in terms of popen3 for windows and @headius happens to have a recent PR #4964 which changes popen3 behavior. Not sure how WIP this PR is but maybe you can try that and see if it fixes anything? (I have not started up Windows to look at this yet).

We probably have a whole class of errors on windows because MRI has a bunch of native code which we have not yet made the equivalent in jnr-posix. This actually expands out to features of IO in general. We would have worked on this but it is a big job.

cshupp1 · 2018-01-29T17:32:46Z

@enebo

How do I get jruby's complete jar?
mvn -Pcomplete?

See this gist:

https://gist.github.com/cshupp1/7916b7027d0163a6b7d8ec32643743bf

cshupp1 · 2018-01-29T18:09:36Z

@enebo
I got it building... I always forget to unset JRUBY_HOME.

irb will not even come up.
https://thepasteb.in/p/WnhzYzywQWQTV

Cris

cshupp1 · 2018-01-29T19:29:26Z

On:

C:\work\jruby\jruby>git branch
* improve_windows_open3
  master

The results are unchanged.

irb(main):004:0> JRUBY_VERSION
=> "9.1.16.0-SNAPSHOT"
irb(main):005:0>
irb(main):001:0> require "open3"
=> true
irb(main):002:0> stout, stderr, status = Open3.capture3({}, "which bundle")
=> ["/c/work/VSO/gem_home/bin/bundle\n", "", #<Process::Status: pid 10692 exit 0>]
irb(main):003:0>  sttdout, stderr, status = Open3.capture3({}, "bundle exec rake routes")
IOError: Cannot run program "bundle" (in directory "C:\work\VSO\vso"): CreateProcess error=2, The system cannot find the file specified
        from org/jruby/RubyIO.java:4847:in `popen3'
        from uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jruby/open3_windows.rb:7:in `popen3'
        from uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:258:in `capture3'
        from (irb):3:in `<eval>'
        from org/jruby/RubyKernel.java:995:in `eval'
        from org/jruby/RubyKernel.java:1316:in `loop'
        from org/jruby/RubyKernel.java:1138:in `catch'
        from org/jruby/RubyKernel.java:1138:in `catch'
        from uri:classloader:/META-INF/jruby.home/bin/jirb:13:in `<main>'
irb(main):004:0>

cshupp1 · 2018-01-29T19:55:44Z

FYI, this does work (Still using improve_windows_open3) (the last one)

irb(main):006:0>  sttdout, stderr, status = Open3.capture3({}, "cmd")
=> ["Microsoft Windows [Version 10.0.16299.192]\r\n(c) 2017 Microsoft Corporation. All rights reserved.\r\n\r\nC:\\work\\VSO\\vso>", "", #<Process::Status: pid 16776 exit 0>]
irb(main):007:0>  sttdout, stderr, status = Open3.capture3({}, "jruby bundle exec rake routes")
=> ["", "Error opening script file: bundle (No such file or directory)\r\n", #<Process::Status: pid 22440 exit 1>]
irb(main):008:0>  sttdout, stderr, status = Open3.capture3({}, "jruby c:/work/vso/gem_home/bin/bundle exec rake routes")
=> ["\"The version is \"\r\n\"Setting rails war to use production\"\r\n    Prefix Verb URI Pattern                      Controller#Action\r\n      root GET  /                                welcome#index\r\nfetch_text GET  /fet
ch_text/fetch_text(.:format) fetch_text#fetch_text\r\n", "C:/work/VSO/vso/config/environment.rb:5: warning: already initialized constant WINDOWS\r\nC:/work/VSO/vso/config/environment.rb:7: warning: already initialized constant
CATALINA_HOME\r\n", #<Process::Status: pid 21152 exit 0>]

cshupp1 · 2018-01-29T20:52:08Z

So for clarity:
stdout, sterr, status = Open3.capture3({}, "bundle.bat exec webpack")
That works on JRUBY
stdout, sterr, status = Open3.capture3({}, "bundle exec webpack")
That fails on jruby

Both work in MRI.

headius · 2018-02-13T14:41:17Z

Open3.capture3({}, "jruby bundle exec rake routes")

This probably should be -S bundle. Wouldn't expect it to work as you show it.

I'll boot up my Windows VM and see what I can see today.

headius · 2018-02-13T16:18:54Z

Ok, so the only problem I see here is that it's not finding bundle.bat, because the following seems to work fine:

>jruby -ropen3 -e "p Open3.capture3({}, 'bundle.bat exec gem list')"
["bundler (1.16.1)\r\n", "", #<Process::Status: pid 9764 exit 0>]

I'll look into the path searching logic, but can you confirm that providing a more specific command makes all your examples work?

headius · 2018-02-13T16:53:02Z

Ok, I've pushed some tweaks to the branch that do the following:

Enable executable-searching for popen3. We did it for other forms of process-launching, but not this one for some reason.
Tweak subshell detection to use cmd.exe when the target executable is a batch or cmd script.

This appears to make your examples all work right.

headius · 2018-02-14T20:43:02Z

Ok, this looks ok in CI and it makes all your examples work, so we'll call it fixed.

cngshow · 2018-03-07T20:20:07Z

@headius
@enebo

Need this reopened, when trying to generate a war with warbler the dependency on webpacker still fails:

IOError: Cannot run program "bundle" (in directory "C:\Users\Sanjay\git\drc"): C
reateProcess error=2, The system cannot find the file specified
C:/Users/Sanjay/git/drc/gem_home/gems/webpacker-3.2.2/lib/webpacker/compiler.rb:
56:in `run_webpack'

The code referenced is:

sterr, stdout, status = Open3.capture3(webpack_env, "bundle exec webpack")

headius · 2018-03-15T18:15:30Z

Is Warbler using JRuby 9.1.16 to generate this war file? It sure looks like it doesn't have my fix.

cngshow · 2018-03-20T18:25:26Z

Warbler does appear to be using 9.1.16.

Consider the following:

@ c:\temp\crap
$ ls
foo.bat
@ c:\temp\crap
$ foo

@ c:\temp\crap
$ echo I love you Cris!
I love you Cris!

Now consider the following irb:

$ irb
irb(main):001:0> JRUBY_VERSION
=> "9.1.16.0"
irb(main):002:0> require 'open3'
=> true
irb(main):003:0> stdout, sterr, status = Open3.capture3({}, 'foo.bat')
=> ["\r\n\e[32m\e]9;8;\"USERNAME\"\e\\@\e]9;8;\"COMPUTERNAME\"\e\\ \e[92mc:\\temp\\crap\e[90m\r\n\e[90m$\e[m echo I love you Cris! \r\nI love you Cris!\r\n", "", #<Process::Status: pid 16604 exit 0>]
irb(main):004:0> stdout, sterr, status = Open3.capture3({}, 'foo')
IOError: Cannot run program "foo" (in directory "c:\temp\crap"): CreateProcess error=2, The system cannot find the file specified
        from org/jruby/RubyIO.java:4892:in `popen3'
        from C:/languages/ruby/jruby-9.1.16.0/lib/ruby/stdlib/jruby/open3_windows.rb:74:in `popen3'
        from C:/languages/ruby/jruby-9.1.16.0/lib/ruby/stdlib/jruby/open3_windows.rb:272:in `capture3'
        from (irb):4:in `<eval>'
        from org/jruby/RubyKernel.java:995:in `eval'
        from org/jruby/RubyKernel.java:1316:in `loop'
        from org/jruby/RubyKernel.java:1138:in `catch'
        from org/jruby/RubyKernel.java:1138:in `catch'
        from C:/languages/ruby/jruby-9.1.16.0/bin/jirb:13:in `<main>'
irb(main):005:0>

That is unexpected in 9.1.16 right?

cngshow · 2018-03-20T20:32:35Z

BTW, the bomb out isn't happening in warbler. It is webpacker.

headius · 2018-03-26T22:21:29Z

Hmm...tracing through the logic for Open3.capture3 in JRuby leads me to popen3, popen_run, and then spawn which on Windows goes through our ShellLauncher logic. That logic in turn does appear to be doing the executable search.

Indicating that we should do a search:

jruby/core/src/main/java/org/jruby/util/ShellLauncher.java

Line 452 in 6e2088a

LaunchConfig cfg = new LaunchConfig(runtime, rawArgs, true);

Proceeding to the search:

jruby/core/src/main/java/org/jruby/util/ShellLauncher.java

Line 1171 in 6e2088a

if (doExecutableSearch && shouldVerifyPathExecutable(cmdline) && !cmdBuiltin) {

The search:

jruby/core/src/main/java/org/jruby/util/ShellLauncher.java

Line 415 in 6e2088a

public static File findPathExecutable(Ruby runtime, String fname) {

There's also conditional logic that disables the executable search if any of several shell characters (/['"<>|%\\]/) appear in it. That does not appear to be the case here.

I should be able to reproduce this on my Windows VM.

headius · 2018-03-26T22:26:24Z

Ok, I don't have a Windows VM yet on this machine. It will take some time to set up. @enebo if you are looking for something to do...

headius · 2018-04-12T19:00:12Z

There were two popen path fixes made on master that never propagated to 9.1: 1925b5d and b0f7aaf. The first uses path searching for popen, and the second fixes a regression in the first (ignoring system PATH).

I've cherry-picked them to 9.1 as 686fca1 and 85cb4e9.

Fingers crossed, I believe this should fix your issue.

enkessler · 2019-02-11T19:10:30Z

I have run into this problem on both 9.1.17 and 9.2.5. It does not seem to be fixed.

enebo · 2019-02-11T19:44:42Z

@enkessler can you make a little reproducer? I tried several of the things above and I got them to work but our windows process launching stuff is complicated so the specifics of the command string could make a huge difference on figuring out what is wrong.

enkessler · 2019-02-13T16:21:42Z

@enebo Here's the reproduction repo. Just bundle install and then bundle exec rake and it should blow up nicely. FYI, I'm using Windows 7 and JRuby 9.2.5

$ jruby -v
jruby 9.2.5.0 (2.5.0) 2018-12-06 6d5a228 Java HotSpot(TM) 64-Bit Server VM 25.202-b08 on 1.8.0_202-b08 +jit [mswin32-x86_64]

enebo · 2019-02-14T16:12:16Z

@enkessler ok I can repro on Win 10. If I change to bundler.bat it works so I feel like we have been here before (in looking at previous comments). Thanks for taking the time to make this.

enebo · 2019-02-15T20:09:54Z

@enkessler Looks like we had some reasonable logic commented out which addresses this problem. If we run from .bat I guess we cannot exec but have to run the command within a CMD shell.

enkessler · 2019-02-15T20:16:08Z

Nice to know that is was an easy fix. Thanks!

cshupp1 mentioned this issue Jan 29, 2018

Webpacker::Compiler does not work on windows rails/webpacker#1175

Closed

enebo added this to the JRuby 9.1.16.0 milestone Jan 29, 2018

headius mentioned this issue Jan 29, 2018

Open3.open2e returns Errno::ENOENT in jRuby but runs fine on Ruby #4943

Closed

headius added a commit that referenced this issue Feb 13, 2018

Perform executable search and use shell for bat/cmd. Fixes #5018.

95b253c

headius closed this as completed Feb 14, 2018

enebo modified the milestones: JRuby 9.1.16.0, JRuby 9.1.17.0 Mar 7, 2018

enebo reopened this Mar 7, 2018

cshupp1 mentioned this issue Apr 11, 2018

From the open3 docs: rails/webpacker#1421

Merged

headius closed this as completed Apr 12, 2018

enkessler mentioned this issue Feb 13, 2019

TEST_ENV_NUMBER is nil for JRuby grosser/parallel_tests#685

Open

enebo reopened this Feb 15, 2019

enebo closed this as completed in 2cd1344 Feb 15, 2019

jcouball mentioned this issue Jan 5, 2021

When extension is omitted, IO.popen does not find the executable in JRuby on Windows #6516

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Sponsors

open3.rb broken in JRuby #5018

open3.rb broken in JRuby #5018

cshupp1 commented Jan 29, 2018 •

edited

Loading

enebo commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

enebo commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

cshupp1 commented Jan 29, 2018 •

edited

Loading

cshupp1 commented Jan 29, 2018 •

edited

Loading

headius commented Feb 13, 2018

headius commented Feb 13, 2018

headius commented Feb 13, 2018

headius commented Feb 14, 2018

cngshow commented Mar 7, 2018

headius commented Mar 15, 2018

cngshow commented Mar 20, 2018

cngshow commented Mar 20, 2018

headius commented Mar 26, 2018

headius commented Mar 26, 2018

headius commented Apr 12, 2018

enkessler commented Feb 11, 2019

enebo commented Feb 11, 2019

enkessler commented Feb 13, 2019

enebo commented Feb 14, 2019

enebo commented Feb 15, 2019

enkessler commented Feb 15, 2019

open3.rb broken in JRuby #5018

open3.rb broken in JRuby #5018

Comments

cshupp1 commented Jan 29, 2018 • edited Loading

enebo commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

enebo commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

cshupp1 commented Jan 29, 2018

cshupp1 commented Jan 29, 2018 • edited Loading

cshupp1 commented Jan 29, 2018 • edited Loading

headius commented Feb 13, 2018

headius commented Feb 13, 2018

headius commented Feb 13, 2018

headius commented Feb 14, 2018

cngshow commented Mar 7, 2018

headius commented Mar 15, 2018

cngshow commented Mar 20, 2018

cngshow commented Mar 20, 2018

headius commented Mar 26, 2018

headius commented Mar 26, 2018

headius commented Apr 12, 2018

enkessler commented Feb 11, 2019

enebo commented Feb 11, 2019

enkessler commented Feb 13, 2019

enebo commented Feb 14, 2019

enebo commented Feb 15, 2019

enkessler commented Feb 15, 2019

cshupp1 commented Jan 29, 2018 •

edited

Loading

cshupp1 commented Jan 29, 2018 •

edited

Loading

cshupp1 commented Jan 29, 2018 •

edited

Loading