Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unskip testdriver Actions infra tests #19497

Merged

Conversation

LukeZielinski
Copy link
Contributor

The tests in infrastructure/testdriver/actions were being skipped due to a Chromedriver bug (https://bugs.chromium.org/p/chromedriver/issues/detail?id=3114). This seems unreproducible with the latest version so I'm re-enabling these tests.

@LukeZielinski
Copy link
Contributor Author

Ok so this is interesting. When running Chrome on MacOS normally, the test actionsWithKeyPressed Fails and the next test (elementPosition) passes. If I add a failure expectation for actionsWithKeyPressed then it fails (and is treated as OK) but then elementPosition times out.

Logs for state 1: No expectation, actionsWithKeyPressed fails unexpected and elementPosition passes.

TEST-START | /infrastructure/testdriver/actions/actionsWithKeyPressed.html
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
TEST-UNEXPECTED-FAIL | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | TestDriver actions: actions with key pressed - assert_array_equals: lengths differ, expected 3 got 1
    at Test. (http://web-platform.test:8000/infrastructure/testdriver/actions/actionsWithKeyPressed.html:64:34)
    at Test.step (http://web-platform.test:8000/resources/testharness.js:1908:25)
    at http://web-platform.test:8000/resources/testharness.js:1948:32
TEST-OK | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | took 2621ms
Closing logging queue
queue closed
PID 1567 | Starting ChromeDriver 79.0.3921.0 (15bace96a14b174529914de372e04fd14c78eac7-refs/heads/master@{#698801}) on port 49748
PID 1567 | Only local connections are allowed.
PID 1567 | Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Starting runner
PID 1567 | [1570131757.641][WARNING]: You are using an unsupported command-line switch: --disable-build-check. Please don't report bugs that cannot be reproduced with this switch removed.
TEST-START | /infrastructure/testdriver/actions/elementPosition.html
TEST-PASS | /infrastructure/testdriver/actions/elementPosition.html | TestDriver actions: element position 
TEST-OK | /infrastructure/testdriver/actions/elementPosition.html | took 1011ms

Logs for state 2: Failure expectation for actionsWithKeyPressed and it fails as expected, but elementPosition times out.

TEST-START | /infrastructure/testdriver/actions/actionsWithKeyPressed.html
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
TEST-FAIL | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | TestDriver actions: actions with key pressed - assert_array_equals: lengths differ, expected 3 got 1
    at Test. (http://web-platform.test:8000/infrastructure/testdriver/actions/actionsWithKeyPressed.html:64:34)
    at Test.step (http://web-platform.test:8000/resources/testharness.js:1908:25)
    at http://web-platform.test:8000/resources/testharness.js:1948:32
TEST-OK | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | took 2791ms
TEST-START | /infrastructure/testdriver/actions/elementPosition.html
TEST-UNEXPECTED-TIMEOUT | /infrastructure/testdriver/actions/elementPosition.html | Waiting on browser:
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 778, in __bootstrap
    raise
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 855, in __bootstrap_inner
    pass
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 758, in run
    del self.__target, self.__args, self.__kwargs
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 312, in _run
    self.result_flag.set()
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 355, in do_testharness
    parent_window = protocol.testharness.close_old_windows()
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 97, in close_old_windows
    pass
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 20, in inner
    return func(self, *args, **kwargs)
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 576, in close
    handles = self.send_session_command("DELETE", "window")
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 516, in send_session_command
    return self.send_command(method, url, body)
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 480, in send_command
    raise err
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/transport.py", line 176, in send
    return Response.from_http(response, decoder=decoder, **codec_kwargs)
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/transport.py", line 191, in _request
    return self.connection.getresponse()
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse
    return response
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 515, in begin
    self.will_close = 1
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 429, in _read_status
    return version, status, reason
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
TEST-INFO took 20040ms

@jgraham
Copy link
Contributor

jgraham commented Oct 4, 2019

Bet you end up with a context menu open or something and that interfers with the next test.

@wpt-pr-bot wpt-pr-bot added webdriver wg-testing_browser wptrunner The automated test runner, commonly called through ./wpt run labels Oct 4, 2019
@LukeZielinski
Copy link
Contributor Author

@jgraham Sure seems that way - the next test times out waiting for close_old_windows. Sounds like you may have run into this before, any advice?

@jgraham
Copy link
Contributor

jgraham commented Oct 4, 2019

Nothing specific, I just remembered it was doing something weird on mac. I can't find a bug to suggest that we had to fix it specially, so I don't know I have anything to offer other than "either leave it disabled on mac or fix chromedriver to close context menus that would interfere with subsequent commands". The latter of which is possibly lots of work; I don't know.

@LukeZielinski
Copy link
Contributor Author

One more bit of context - the reason this behaviour is manifesting when the failure expectation is added seems to be related to whether we restart chromedriver in between these tests.

In State 1 (No expectation, actionsWithKeyPressed fails unexpected and elementPosition passes) Chromedriver is restarted between the two tests because the first failure is unexpected.

In State 2 (Failure expectation for actionsWithKeyPressed and it fails as expected, but elementPosition times out) the first failure is expected so Chromedriver is not restarted, thus causing the timeout in the 2nd test.

@jugglinmike
Copy link
Contributor

The WPT CLI's --no-restart-on-unexepected option might help with debugging

@LukeZielinski
Copy link
Contributor Author

Suspicion confirmed - one of the tests sends a Ctrl+Click action which opens a context menu on Mac. Test will be updated.

@LukeZielinski
Copy link
Contributor Author

LukeZielinski commented Oct 9, 2019

The offending test has been fixed (crrev.com/c/1845994), this PR starts running at again. @foolip: PTAL?

@foolip
Copy link
Member

foolip commented Oct 9, 2019

Sweet, thanks @LukeZielinski!

@foolip foolip merged commit 8f21bdd into web-platform-tests:master Oct 9, 2019
@LukeZielinski LukeZielinski deleted the unskip-actions-infra-test branch November 22, 2019 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra webdriver wg-testing_browser wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants