@jgraham

All but one seems to be an actual stability check failure, so if there's a problem here it isn't with Taskcluster as such, but with the stability checking.

The trouble would be if we have so much pre-existing flakiness that an unreasonable number of PRs would be blocked on pre-existing flakiness. I don't know where to draw the line, filed this issue to start collecting cases of it happening.

If it's too bad, I think we'd have to get our hands on data about which tests are most frequently affected and which of those are most flaky, and fix those.

Fixing flaky tests seems like a good idea for many reasons.

@lukebjerring @mdittmer for any given test that shows up as flaky in these PRs, is there a straightforward way to get a good guess answer to "was it already flaky?" from wpt.fyi? data? The historical red/green boxes hidden behind a button on wpt.fyi still interleave different versions so it's hard to tell why a test is alternating between passing and failing.

For #14551, I've restarted it a couple of times by pushing a rebased commit, and I believe that it was once reported as firefox flakiness, and twice as Chrome's. But I've never been able to reproduce locally, and while instability in a basic non animated layout reftest isn't totally impossible if the browser engine happens to have some kind of non deterministic bug, it is pretty unlikely. So I don't know what's causing this, but I'm pretty sure it isn't the test itself.

Self-assigning so that at least I'll check in later to see how many PRs are blocked on stability.

cc @Hexcles

Yes, we can now somewhat-reliably see existing flakiness in the last ten runs in staging:
https://staging.wpt.fyi/insights

This keeps being the main time sink when monitoring https://ecosystem-infra-rotation.appspot.com/, showing up as "blocked export PRs."

@lukebjerring @KyleJu do you think anything beyond the currently planned work on flakiness metadata will be needed to resolve this? More concretely, will the flakiness still show up as blocked exports and require us to add the metadata, or do we need to block Chromium CLs from landing? There's probably a tricky balance here between the amount of human involvement on our side vs. how much overhead there is for Chromium devs trying to get stuff landed and just using WPT because it's the default thing to do. Disincentives for using WPT would be bad :)

Closing this, work is underway to not block on known flakiness.