New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[testharness.js] Don't complete after allowed exc. #19612
[testharness.js] Don't complete after allowed exc. #19612
Conversation
The following tests intentionally produce an uncaught exception and then define subtests: - html/semantics/scripting-1/the-script-element/module/error-and-slow-dependency.html - html/webappapis/scripting/processing-model-2/window-onerror-parse-error.html - html/webappapis/scripting/processing-model-2/window-onerror-runtime-error.html - html/webappapis/scripting/processing-model-2/window-onerror-runtime-error-throw.html testharness.js immediately transitions to "complete" and ignores the subsequent subtests.
This patch caused a failure in the testharness.js test suite, and that failure demonstrated an edge case I hadn't considered. After looking in to it further (research summarized below), it looks like the edge case was limited to the infrastructure test itself. I don't think it demonstrates anything unsound about this patch. It's possible that worker-based tests (which require It doesn't look like this is a problem, though. There are six worker sources which enable
With the exception of the testharness.js test mentioned above, all of them invoke There are three multi-global tests that use
For these, the |
In principle this looks OK, I think. But please push to at least one of the |
So I see a Fx run; the wpt.fyi diff is https://wpt.fyi/results/?diff&filter=ADC&run_id=321040007&run_id=344290004 Are any of those changes worrying? |
@jgraham Do you think this needs further validation? |
The following tests intentionally produce an uncaught exception and then define
subtests:
testharness.js immediately transitions to "complete" and ignores the
subsequent subtests.
Today, the tests referenced in the commit message are reported as passing single-page tests. Once we've completed the implementation of WPT RFC 28, they will instead be reported as test harness errors.
As an alternative to this patch, we could modify each of the tests. If a subtest is defined before the exception is thrown, then all the subtests are reported as expected. (We could, for example, declare the existing synchronous subtests as asynchronous subtests instead.)
Changing the harness is definitely riskier, but it also seems more correct to me.
This change could interfere with single-page tests that use
allow_uncaught_exceptions
. My experiments don't indicate that any such tests exist, but it's difficult to demonstrate or summarize that research here. A more blunt heuristic might do. Only three tests useallow_uncaught_exception
without invoking functions namedtest
,promise_test
, orasync_test
:All three define subtests indirectly through helper functions.