[offscreen-canvas] Correctly report result of asynchronous operations #20277
Conversation
Update the templates and test body code to correctly track the status of asynchronous operations. Remove invocations of the `deferTest` utility function because it has no effect in this context.
wpt-pr-bot
requested review from
AmeliaBR,
annevk,
fserb,
gsnedders and
kenrussell
|
I can't help wondering if we should have different templates for synchronous vs. asynchronous tests, just to avoid so much duplication. |
I wondered that, too. I decided against it because I saw that the test files are hard-coded, but now I can see that there's a specific provision for this test suite. I'll see what I can do about this... |
|
@jdm It looks like using multiple templates will require more effort than extending the list in that script. I won't have the bandwidth to do that any time soon, but I would like to see these tests made technically correct. Would you be willing to tolerate the duplication as technical debt? |
|
Yes, I won't insist on that change when it's holding back actual test correctness issues. |
|
Note, dce0fc9#diff-8bf796e810f7184850d920e92c11bad2R46 currently will throw an exception at Firefox and Nightly whether the test is synchronous or asynchronous (#19799). |
|
Note also, the messages printed at https://wpt.fyi/results/offscreen-canvas/the-offscreen-canvas/2d.getcontext.exists.html?diff&filter=ADC&run_id=358480001&run_id=343560004 for Firefox
and
are incorrect. |
|
Is the Firefox version being tested launched with |
Currently Firefox requires the preference If no preferences were passed to Firefox being used the above quote would be correct and the test re Firefox moot, however, the browser used by WPT does pass custom preferences to the browser instance web-platform-tests/wpt.fyi#1648 (comment). Therefore |
|
@guest271314 Thanks, that's all good information. This pull request is focused on correcting the tests themselves, so discussion about configuring the browsers is probably better left to a separate issue. |
|
The stability check in Firefox appears to be timing out due to the number of tests modified by this patch. @jdm if you could verify my interpretation, then I will override the warning about the failed job. |
|
Agreed! |
This resolves gh-20180.
In
mastertoday, the test results are misrepresenting implementation status. Many failing tests are incorrectly reporting as having passed. Some of them also report an error in the testing harness (which ought to hint that something is going wrong), but many claim to beOK.This patch makes the tests accurately report
PASSorFAILand removes the test harness errors.To demonstrate this, I scheduled trials for the patch in Chrome and Firefox. We can use https://wpt.fyi to compare the results reported for each browser with and without this patch (hint: there is a toggle button in the UI just below the icon labeled "diff" that helps you to view only differences):
Firefox does not implement OffscreenCanvas, so (as expected) this patch had no effect on Firefox's results. There is one discrepancy, but that's due to an unrelated crash--something that happens from time to time in the test environment.
Chrome's results are much improved. Individual test results changed in one of two ways, as reported in the wpt.fyi UI:
+1 / -1 / 0- this means there is one more passing "statistic" and one more failing "statistic." The statistics combine subtest results and harness status, so this specifically means: the harness status changed fromERRORtoOKand the subtest result changed fromPASStoFAIL.0 / -1 / 0- here, the harness status was unchanged, and the subtest result changed fromPASStoFAIL.I've split this into two commits: one for the template and source file changes, and one for the corresponding changes to the generated test material. I don't know how you folks prefer to review or merge changes like this, though, and I'm happy to reconfigure as you please.