New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integration of CTA WAVE Test Suite for Embedded Devices with WPT #16214
Comments
How does the Node.js server control DUT? The graph seems to suggest that 13 is the response of request 10. If so, when DUT receives the new URL, how does it navigate to there? And what happens if there are errors and the browser on DUT hangs? |
The Node.js server has no direct control over the DUT. To avoid manipulating the original tests, we added some calls to the testharnessreport.js. When a result is registered by the testharnessreport.js, the same data is sent to the Node.js server using an ajax POST request. Then, in a separate ajax request, we ask the Node.js server what the next test is. The server sends back the url to the next test, which is then navigated to by calling location.href = . Alternatively, if this method fails, the result data is uri encoded and sent via GET request by setting the location.href. In this case the next test is navigated to by using a redirect from the Node.js server. For error handling we depend on the built-in functionality of the test and the testharness to then again call logic for sending the results from the testharnessreport.js. If the testharness does not recognize an error and the test is stuck, a timeout (default 60s) is set to automatically send a timeout result to the server and navigate to the next test. If the browser crashes, there is not much we can do about that, so in the scope of the WAVE project we filter out those tests by testing reference browsers. Should the test run hang nonetheless, the session is kept by the server, so everything you need to resume the run with the next test is to remember the session token and requesting the corresponding html page. |
+1 These proposed extensions allow W3C Web platform tests to be executed on Smart TVs. These devices constitute one of the largest playback surfaces of OTT content, especially UHD content. TV API support is historically weak compared to laptops, desktops and mobile phones. To move web forward to the point where it provides the de facto common media playback environment across all devices, we need tests which can execute across all devices. This proposed change is a logical step forward in improving the reach of WP tests. Will Law |
WPT is great for downloadable browsers on e.g. laptops. It can be used on embedded devices (e.g., smart TVs, STBs, media sticks) as well. But WPT-on-embedded requires a lot of work to port WPT and make it easy to use. These extensions for embedded devices are being developed and maintained through the CTA WAVE Project. What should happen next? These embedded tests should be merged with the main WPT effort, now and as a going-forward effort by the WPT editors and the CTA WAVE team. The WAVE leadership made a clear decision to NOT permanently fork WPT. But this decision isn't entirely up to WAVE. If the proposal in this Issue is approved, WPT will gain the embedded devices supported needed for TVs and STBs. CTA WAVE plans to continue to support and extend the embedded extensions and hopes to continue working with the WPT editors on enhancements to improve HTML5 video interoperability across the broad spectrum of devices. Comments welcome on the appropriateness of this approach, thanks. s/Mike Bergman (About CTA WAVE, for those not involved: WAVE is a broad industry effort on content, HTML5 APIs and device performance to improve interoperability of commercial internet video for HTML5 UA devices and for other frameworks. Working in conjunction with the W3C Web Media API Community Group, WAVE has developed an HTML5 framework interoperability specification that cites W3C specification for all HTML-related requirements. See CTA.tech/WAVE for more information on the WAVE Project, including how to participate, or dm me.) |
After skimming the comments here, a few high-level questions (sorry if they're answered already):
|
So my understanding is that this is for devices where remote control via webdriver or similar isn't possible. I think that's important, because the wptrunner code as of recently allow you to run in a single window (or could be easily modified to do so), so if it's about devices we can remote control but which can't open a new window, I'd say we can use that. Assuming that assumption is correct, I'm sure it could fit into the wptrunner architecture, but it's not a 1:1 mapping to anything that already exists. That said, there are some advantages to integrating at that point, for example the support for test expectation data. Taking the proposal on its own terms, I think
|
Thanks for the replies. I think I have a better understanding of this now. First of all, I can see the motivation behind this feature and it's potentially also useful in other scenarios (e.g. mobile testing on iOS where automation isn't easy). I have some questions/concerns regarding some technical details:
|
reply to @Ms2ger comments
Yes. Exception: If the browser on the embedded device crashes, you can continue the last session after a manual restart. The test that crashed the browser will be skipped.
Yes
In our case, we are considering devices like TVs and STBs without webdriver support. This is why 'wptrunner' is not an option for us.
Yes, we will create a PR soon.
We can do it if there is a need. |
Thanks @jgraham for your feedback. I made a mistake in the architecture diagram above we only updated |
Thanks @Hexcles for your comments.
Agree.
Correct If the browser on the DUT crashes you need to restart manually but you can continue the last session and the test that caused the crash will be automatically skipped. The number of crashes depends on the DUT. For example, on a TV set from 2017 or 2018 we may need to exclude around 5-10 tests. |
All, we created a PR to the RFC GitHub Repository as you recommended. Looking forward to your support and get the new features integrated into WPT. |
This was done in web-platform-tests/rfcs#23. |
The Web Media API Test Suite is a fork of the WPT Test suite to cover embedded devices like TVs and STBs which are not supported by the current WPT Test Runner. The main limitation of the WPT test runner on embedded devices is that it requires multiple browser windows, one for the test runner itself and one for running individual tests as depicted in the high-level architecture below:
The Web Media API Test Suite extends the WPT Runner in a way that only a single window is required to run the tests. The Test Runner itself is migrated to a Node.js Server as depicted in the overall architecture of the Web Media API Test Suite below. The embedded device runs individual tests and sends the results to the Node.js Test Runner via XHR which update the internal state of the running test session and pushes the new updates to a companion page which shows the progress of a test session (The Companion Page offers also other helpfull features like generating test reports, exporting test results, compare test sessions, etc. through an easy to use interface). Once a Test is completed, the Node.js Test Runner sends the URL of the next test which will be opened in the same browser window and the steps will be repeated until all tests are completed. The WPT test files are not changed we only extended
testharness.js
andtestharnessreport.js
to enable the integration with the Node.js Test Runner. These are the main two files we extended.We had a discussion during last W3C TPAC in France about integrating the new Test Suite in WPT to support embedded devices and this issue is about to discuss with WPT folks about concrete next steps. The last version of the Web Media API Test Suite is avaible in this branch https://github.com/cta-wave/WMAS/tree/wmas2018 which is a fork of WPT from december 2018.
The text was updated successfully, but these errors were encountered: