Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration of CTA WAVE Test Suite for Embedded Devices with WPT #16214

Closed
louaybassbouss opened this issue Apr 2, 2019 · 12 comments
Closed
Labels
infra priority:backlog wptrunner The automated test runner, commonly called through ./wpt run

Comments

@louaybassbouss
Copy link
Contributor

The Web Media API Test Suite is a fork of the WPT Test suite to cover embedded devices like TVs and STBs which are not supported by the current WPT Test Runner. The main limitation of the WPT test runner on embedded devices is that it requires multiple browser windows, one for the test runner itself and one for running individual tests as depicted in the high-level architecture below:

current-wpt-testrunner

The Web Media API Test Suite extends the WPT Runner in a way that only a single window is required to run the tests. The Test Runner itself is migrated to a Node.js Server as depicted in the overall architecture of the Web Media API Test Suite below. The embedded device runs individual tests and sends the results to the Node.js Test Runner via XHR which update the internal state of the running test session and pushes the new updates to a companion page which shows the progress of a test session (The Companion Page offers also other helpfull features like generating test reports, exporting test results, compare test sessions, etc. through an easy to use interface). Once a Test is completed, the Node.js Test Runner sends the URL of the next test which will be opened in the same browser window and the steps will be repeated until all tests are completed. The WPT test files are not changed we only extended testharness.js and testharnessreport.js to enable the integration with the Node.js Test Runner. These are the main two files we extended.

single-window-wpt-testrunner

We had a discussion during last W3C TPAC in France about integrating the new Test Suite in WPT to support embedded devices and this issue is about to discuss with WPT folks about concrete next steps. The last version of the Web Media API Test Suite is avaible in this branch https://github.com/cta-wave/WMAS/tree/wmas2018 which is a fork of WPT from december 2018.

@gsnedders gsnedders added infra wptrunner The automated test runner, commonly called through ./wpt run labels Apr 9, 2019
@Hexcles
Copy link
Member

Hexcles commented Apr 12, 2019

How does the Node.js server control DUT? The graph seems to suggest that 13 is the response of request 10. If so, when DUT receives the new URL, how does it navigate to there? And what happens if there are errors and the browser on DUT hangs?

@squapp
Copy link
Contributor

squapp commented Apr 24, 2019

The Node.js server has no direct control over the DUT. To avoid manipulating the original tests, we added some calls to the testharnessreport.js. When a result is registered by the testharnessreport.js, the same data is sent to the Node.js server using an ajax POST request. Then, in a separate ajax request, we ask the Node.js server what the next test is. The server sends back the url to the next test, which is then navigated to by calling location.href = . Alternatively, if this method fails, the result data is uri encoded and sent via GET request by setting the location.href. In this case the next test is navigated to by using a redirect from the Node.js server.

For error handling we depend on the built-in functionality of the test and the testharness to then again call logic for sending the results from the testharnessreport.js. If the testharness does not recognize an error and the test is stuck, a timeout (default 60s) is set to automatically send a timeout result to the server and navigate to the next test. If the browser crashes, there is not much we can do about that, so in the scope of the WAVE project we filter out those tests by testing reference browsers.

Should the test run hang nonetheless, the session is kept by the server, so everything you need to resume the run with the next test is to remember the session token and requesting the corresponding html page.

@wilaw
Copy link

wilaw commented May 3, 2019

+1

These proposed extensions allow W3C Web platform tests to be executed on Smart TVs. These devices constitute one of the largest playback surfaces of OTT content, especially UHD content. TV API support is historically weak compared to laptops, desktops and mobile phones. To move web forward to the point where it provides the de facto common media playback environment across all devices, we need tests which can execute across all devices. This proposed change is a logical step forward in improving the reach of WP tests.

Will Law
Chair - WAVE Technical Working Group

@mbergman42
Copy link

WPT is great for downloadable browsers on e.g. laptops. It can be used on embedded devices (e.g., smart TVs, STBs, media sticks) as well. But WPT-on-embedded requires a lot of work to port WPT and make it easy to use. These extensions for embedded devices are being developed and maintained through the CTA WAVE Project.

What should happen next? These embedded tests should be merged with the main WPT effort, now and as a going-forward effort by the WPT editors and the CTA WAVE team.

The WAVE leadership made a clear decision to NOT permanently fork WPT. But this decision isn't entirely up to WAVE. If the proposal in this Issue is approved, WPT will gain the embedded devices supported needed for TVs and STBs. CTA WAVE plans to continue to support and extend the embedded extensions and hopes to continue working with the WPT editors on enhancements to improve HTML5 video interoperability across the broad spectrum of devices.

Comments welcome on the appropriateness of this approach, thanks.

s/Mike Bergman
CTA WAVE Project Manager
Vice President, Technology and Standards
Consumer Technology Association
mbergman@CTA.tech

(About CTA WAVE, for those not involved: WAVE is a broad industry effort on content, HTML5 APIs and device performance to improve interoperability of commercial internet video for HTML5 UA devices and for other frameworks. Working in conjunction with the W3C Web Media API Community Group, WAVE has developed an HTML5 framework interoperability specification that cites W3C specification for all HTML-related requirements. See CTA.tech/WAVE for more information on the WAVE Project, including how to participate, or dm me.)

@Ms2ger
Copy link
Contributor

Ms2ger commented May 3, 2019

After skimming the comments here, a few high-level questions (sorry if they're answered already):

@jgraham
Copy link
Contributor

jgraham commented May 3, 2019

So my understanding is that this is for devices where remote control via webdriver or similar isn't possible. I think that's important, because the wptrunner code as of recently allow you to run in a single window (or could be easily modified to do so), so if it's about devices we can remote control but which can't open a new window, I'd say we can use that.

Assuming that assumption is correct, I'm sure it could fit into the wptrunner architecture, but it's not a 1:1 mapping to anything that already exists. That said, there are some advantages to integrating at that point, for example the support for test expectation data.

Taking the proposal on its own terms, I think

  • Having a way to run on devices that don't support automation seems useful
  • Using a custom testharnessreport.js seem totally reasonable; that file is explictly a point of customisation.
  • Having a custom testharness.js is more worrying, and changes to that file should either be moved or integrated with upstream.
  • The pull-based test-loading architecture ("browser fetches the next test, and navigates to it") seems reasonable given the constraints.
  • Having a whole seperate node.js stack running a server when we already have a server written in Python that we have to use anyway seems like unnecessary extra moving parts that will lead to long-term maintainance burden.
  • In principle landing this in wpt seems like a good idea, but would need an RFC and a clear ownership story.

@Hexcles
Copy link
Member

Hexcles commented May 3, 2019

Thanks for the replies. I think I have a better understanding of this now.

First of all, I can see the motivation behind this feature and it's potentially also useful in other scenarios (e.g. mobile testing on iOS where automation isn't easy).

I have some questions/concerns regarding some technical details:

  1. We don't really want to have a whole new language + toolchain (Node) in the mostly Python-based codebase. WPT is deeply integrated into the build systems of at least Chromium & Gecko, and introducing Node dependency would be a complicated task. Besides, I think the server can totally be written in Python.
  2. I asked a few questions about error handling, and if I understand the answers correctly, there isn't really a great way to restart the browser on DUT if things get very wrong. How often do you need to manually intervene in a complete run (or how many tests do you exclude because of crashes)?

@louaybassbouss
Copy link
Contributor Author

reply to @Ms2ger comments

After skimming the comments here, a few high-level questions (sorry if they're answered already):

  • Is this completely automated?

Yes. Exception: If the browser on the embedded device crashes, you can continue the last session after a manual restart. The test that crashed the browser will be skipped.

Yes

In our case, we are considering devices like TVs and STBs without webdriver support. This is why 'wptrunner' is not an option for us.

  • Can you show the difference between your fork and upstream?

Yes, we will create a PR soon.

We can do it if there is a need.

@louaybassbouss
Copy link
Contributor Author

So my understanding is that this is for devices where remote control via webdriver or similar isn't possible. I think that's important, because the wptrunner code as of recently allow you to run in a single window (or could be easily modified to do so), so if it's about devices we can remote control but which can't open a new window, I'd say we can use that.

Assuming that assumption is correct, I'm sure it could fit into the wptrunner architecture, but it's not a 1:1 mapping to anything that already exists. That said, there are some advantages to integrating at that point, for example the support for test expectation data.

Taking the proposal on its own terms, I think

  • Having a way to run on devices that don't support automation seems useful
  • Using a custom testharnessreport.js seem totally reasonable; that file is explictly a point of customisation.
  • Having a custom testharness.js is more worrying, and changes to that file should either be moved or integrated with upstream.
  • The pull-based test-loading architecture ("browser fetches the next test, and navigates to it") seems reasonable given the constraints.
  • Having a whole seperate node.js stack running a server when we already have a server written in Python that we have to use anyway seems like unnecessary extra moving parts that will lead to long-term maintainance burden.
  • In principle landing this in wpt seems like a good idea, but would need an RFC and a clear ownership story.

Thanks @jgraham for your feedback. I made a mistake in the architecture diagram above we only updated testharnessreport.js and NOT testharness.js :). What is your proposal to proceed with the RFC and what are the next steps for us?

@louaybassbouss
Copy link
Contributor Author

Thanks @Hexcles for your comments.

Thanks for the replies. I think I have a better understanding of this now.

First of all, I can see the motivation behind this feature and it's potentially also useful in other scenarios (e.g. mobile testing on iOS where automation isn't easy).

I have some questions/concerns regarding some technical details:

  1. We don't really want to have a whole new language + toolchain (Node) in the mostly Python-based codebase. WPT is deeply integrated into the build systems of at least Chromium & Gecko, and introducing Node dependency would be a complicated task. Besides, I think the server can totally be written in Python.

Agree.

  1. I asked a few questions about error handling, and if I understand the answers correctly, there isn't really a great way to restart the browser on DUT if things get very wrong. How often do you need to manually intervene in a complete run (or how many tests do you exclude because of crashes)?

Correct If the browser on the DUT crashes you need to restart manually but you can continue the last session and the test that caused the crash will be automatically skipped. The number of crashes depends on the DUT. For example, on a TV set from 2017 or 2018 we may need to exclude around 5-10 tests.

@louaybassbouss
Copy link
Contributor Author

louaybassbouss commented Jun 11, 2019

All, we created a PR to the RFC GitHub Repository as you recommended. Looking forward to your support and get the new features integrated into WPT.

@foolip
Copy link
Member

foolip commented May 6, 2021

This was done in web-platform-tests/rfcs#23.

@foolip foolip closed this as completed May 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra priority:backlog wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

No branches or pull requests

9 participants