@boris-petrov Can you narrow down what it is we're doing wrong within Watir and its dependencies? Perhaps at the point of the error, compare the request/etc with what you'd see in MRI?

@headius - thanks for the answer! I don't think I'll have the time to debug this any time soon. The steps I've given above are simple to reproduce so hopefully someone can try figuring it out. It's not urgent at all, it's just something to be fixed at some point. Sorry I can't be of more help right now and thanks for the hard work on JRuby!

Hmm, I got a different result. What did I do wrong?

$ jruby blah.rb
2018-01-25 12:00:48 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:00:48 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:00:49 DEBUG Webdrivers Versions now located on downloads site: []
StandardError: Unable to find the latest version of geckodriver; try downloading manually from https://github.com/mozilla/geckodriver/releases and place in /Users/headius/.webdrivers
       update at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/webdrivers-3.2.4/lib/webdrivers/common.rb:19
  driver_path at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/webdrivers-3.2.4/lib/webdrivers/selenium.rb:13
   initialize at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/firefox/marionette/driver.rb:37
          new at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/firefox/driver.rb:31
          for at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/common/driver.rb:52
          for at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver.rb:84
   initialize at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/watir-6.10.2/lib/watir/browser.rb:48
       <main> at blah.rb:4

I followed the instructions in the error, and I still get a different result:

$ jruby blah.rb
2018-01-25 12:02:58 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:02:58 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:02:59 DEBUG Webdrivers Versions now located on downloads site: []

And that's it.

@headius - thanks for trying! Now I also receive a different result from the one I pasted before. But in any case the result from MRI is different from the one with JRuby - MRI says:

irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-26 13:47:51 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:47:52 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:47:54 DEBUG Webdrivers Versions now located on downloads site: [0.191, 0.19, 0.18, 0.17, 0.161, 0.16, 0.15, 0.14, 0.13, 0.12]
2018-01-26 13:47:54 DEBUG Webdrivers Checking current version
2018-01-26 13:47:54 DEBUG Webdrivers File is already downloaded: true
2018-01-26 13:47:54 DEBUG Webdrivers Current version of /home/boris/.webdrivers/geckodriver is geckodriver 0.19.1

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

=> #<Watir::Browser:0x7d1eda28bc5bf2ee url="about:blank" title="">

And JRuby says:

irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-26 13:46:36 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:46:37 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:46:39 DEBUG Webdrivers Versions now located on downloads site: []
=> #<Watir::Browser:0x51dd1058 url="about:blank" title="">

Which doesn't make sense as the commands were exactly the same (the ones from my original post). You can reproduce that same thing, yes?

@boris-petrov Yeah I get the same result. I'm not sure what to look into though...I have no idea what any of this output even means. Perhaps there's someone from watir we can get involved here to help us narrow down what's different? There's a lot of code here.

I don't have an idea either unfortunately. Should I open an issue in watir? Not sure how to approach this.

The idea of webdrivers is that it downloads the latest available version of a Selenium driver from the corresponding website. I'm not sure which of those lines actually does that though.

@boris-petrov Opening a bug with watir would be a good place to start. We can make it clear we don't think it's their bug, but we need help diagnosing the actual problem.

@boris-petrov can you throw a debug point in your code and see what the output is of this: https://github.com/titusfortner/webdrivers/blob/master/lib/webdrivers/common.rb#L69

Webdrivers is calling http.get_response on https://github.com/mozilla/geckodriver/releases, and then using nokogiri to parse out the values of the versions found on that page.

My guess is that jruby is returning something different with that call than mri so it isn't finding the same thing to parse, but that's the information that is needed to debug it.

@titusfortner - sorry for the late response. I'll try this next week as I had my hands full this one.

@titusfortner Thanks for the assistance!

@titusfortner, @headius - sorry for the delay. The response thing is exactly the same in both MRI and JRuby - that is, the response's body is the same. So I guess the issue is somewhere later when that body is parsed. Any further ideas? Actually you could try it out yourselves with the steps from the first post - it's pretty easy to reproduce.

@boris-petrov Is it easy to isolate "when that body is parsed" and narrow it down that way? I'm not familiar with the code. I have been able to reproduce but got a bit lost trying to sort out why the bug happens.

@headius yeah, whatever the output of the response is, it's just getting parsed it with Nokogiri here: https://github.com/titusfortner/webdrivers/blob/master/lib/webdrivers/geckodriver.rb#L26.

Actually, if the response is the same, is it possible that this case statement takes a different path between ruby implementations? Should I have a different conditional in there?

@titusfortner I wouldn't expect that to be different. More likely the XML parsing is producing slightly different results so webdrivers can't fine the URL it's looking for.

I think the next step is to figure whether the response content is the same. If it is, then it's an issue of parsing in nokogiri, and we should be able to just proceed from the point in the code you highlighted.

@boris-petrov looks like I was trying to parse the geckodriver page with XML instead of HTML. Please try with 3.4.3.

@titusfortner - thanks for that! With the new version it seems to be working fine although I cannot use it for now to run our integration tests due to #5209. @headius - is there a timeline on a new release? It's been a few months already. :)

I'll leave you to decide whether to close the issue but the inconsistency is still there with the old version of webdrivers (which is some other bug in JRuby, probably minor?).

Thank @twalpole for this fix, glad it is working for you now. I have no idea if it is a jruby or nokogiri issue causing discrepancy compared to mri.

@boris-petrov ASAP for release. We are kinda stuck on a couple blockers like #5333.

Sounds like there wasn't anything to fix in JRuby for this, so I'm closing it.

@headius The issue in webdrivers was fixed by telling Nokogiri to parse as HTML (which the document was) however that doesn't change the fact that when Nokogiri was parsing the document as XML the results were different between MRI and JRuby - so there is still something to fix there.

@twalpole Is there a Nokogiri issue for that? It still wouldn't be a JRuby fix, but I suspect it's due to the Nokogiri JRuby logic using a more strict XML library than libxml (i.e. it's probably rejecting or mutilating bad HTML more than libxml does, producing different output).

If this can be reduced to a simple test case (like a script and some HTML/XHTML content), I'd recommend reporting it to Nokogiri.