Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update chromium logger to identify subtest failures. #21763

Conversation

LukeZielinski
Copy link
Contributor

This also fixes a bug where a subtest failure would be hidden because
the expected_status was evaluated too late in test_end. For a
subtest failure the test status is Pass. But for Chromium, we flip this
"actual" status to a failure.
But the actual status is also used as a fall-back for the expected status
(see explanation in chromium.py:126). So if expected is evaluated after the
"flip", it will just match actual and the failure will be marked as
an "expected" failure.

This also fixes a bug where a subtest failure would be hidden because
the expected_status was evaluated too late in `test_end`. For a
subtest failurei the test status is Pass. But for Chromium, we flip this
"actual" status to a failure.
But the actual status is also used as a fall-back for the expected status
(see explanation in chromium.py:126). So if "expected" is evaluated after the
"flip", it will just match "actual" and the failure will be marked as
an "expected" failure.
@wpt-pr-bot wpt-pr-bot added infra wptrunner The automated test runner, commonly called through ./wpt run labels Feb 12, 2020
@LukeZielinski
Copy link
Contributor Author

@KyleJu FYI, this was carved off from crrev.com/c/2025694 so it's seen some exercise already.

@KyleJu
Copy link
Contributor

KyleJu commented Feb 13, 2020

@LukeZielinski I see one test failure in TaskCluster, ::Sample HTML5 API Tests <- tests/functional/api-tests-1.html FAILED [ 15%]. Not sure if its related?

@LukeZielinski
Copy link
Contributor Author

Doesn't seem related. Having trouble running those tests locally, but lets retry here.

@LukeZielinski
Copy link
Contributor Author

Azure failure seems unrelated, retrying.

Copy link
Contributor

@KyleJu KyleJu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LukeZielinski LukeZielinski merged commit 22a7994 into web-platform-tests:master Feb 14, 2020
@LukeZielinski LukeZielinski deleted the chromium-logger-subtest-fails branch February 19, 2020 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants