Oops, I did not realize uploading a PR would assign reviewers and fire the CI automatically. I'll fix the python tests and lint, then report back.

cc @jcjones

It'd be helpful to Mozilla if we could keep the existing (manual) tests while also adding these VirutalDriver-based auto-tests, until we have parity on VirtualDriver.

This is ready for review @jgraham. The tests pass on chrome 79.0.3941.4 onward -- there were some minor changes to the implementation of the API on 79 that are necessary.

@LukeZielinski can you review this and see if there's anything we'd need to do in Chromium?

@foolip This looks good from my end. However, the new infra test that was added here will be failing on Chrome until v79 rolls up into stable. Are we ok submitting this before that happens? This PR will need to be force-merged anyway since stability checks are timing out.

@jcjones what do you think about temporarily adding a default implementation of test_driver.add_virtual_authenticator(...) that does nothing on firefox instead of failing, so we can run the same tests manually and avoid maintaining two test suites?

@foolip This looks good from my end. However, the new infra test that was added here will be failing on Chrome until v79 rolls up into stable. Are we ok submitting this before that happens? This PR will need to be force-merged anyway since stability checks are timing out.

Yeah it's fine if infra tests are failing as long as it's just due to a missing implementation and it doesn't point to lots of other tests failing for spurious reasons.

@foolip can we merge this now? We can take care of the outstanding comment in a follow-up, if needed.

Sorry for the delay. I see that "infrastructure/ tests" is still failing on Taskcluster. @LukeZielinski can you help with how to update the expectations to make this pass?

@nsatragno FYI the infrastructure/ tests aren't expected to always pass on all browsers, but we maintain expectations for them (unlike the rest of WPT) so that regressions that would affect lots of tests don't go unnoticed.

@stephenmcgruer FYI, this is both an example of the sort of test automation addition that we need to be smooth and not double work to make work in Chromium and WPT, and it also illustrates why we need to get a handle on the CI setup in WPT – it's hard to understand failures in Taskcluster and not always clear which tasks we might accept to fail and admin merge.

@nsatragno FYI the infrastructure/ tests aren't expected to always pass on all browsers, but we maintain expectations for them (unlike the rest of WPT) so that regressions that would affect lots of tests don't go unnoticed.

@foolip Earlier in this thread (here) it sounded like we were ok with the failing infra test because it would fix itself when Chrome 79 showed up. We could make it pass via the expectations in this PR, but then we'd have to change them again for Chrome 79. So I was leaning towards just letting this go through.

That it fails is OK, but expectations have to match or every PR touching tools/ and resources/ will fail these tests and need to be admin merged.

When Chrome 79 is released the expectations will be wrong and PRs will start failing, see #14456 for the issue and #19386 for a suggested solution.

Looks like we took so long that the version of chrome running on taskcluster passes now! :)

Looks like the new failures are unrelated? 🤔 @foolip

What remains is wpt-firefox-nightly-stability and wpt-chrome-dev-stability timing out because there were too many affected tests (397) and they couldn't run 10 times in the 2 hours allowed. I'll admin merge.

This was imported into Chromium and looks like it doesn't pass there - expected? https://crbug.com/1023737

@nsatragno @LukeZielinski