[Taskcluster] Switch to repeat-only for stability checks #25149
Conversation
stephenmcgruer
force-pushed
the
smcgruer/stability-checks-repeat-onky
branch
from
8543e39
to
a1722bd
Compare
We frequently hear from test authors that they find it frustrating when the stability checks turn up failures that they cannot reproduce. One common case comes from the fact that stability checks run the same test repeatedly without restarting the browser (the AAABBBCCC behavior). This is unlike any other way we run tests, and can cause some tests to consistently appear flaky due to global state (e.g. the origin-isolation/ tests). To fix this, switch the stability checks to only use 'repeat-restart' flake detection (previously we used both 'repeat-loop' and 'repeat-restart'). This mode run the tests in entire sets, then restarts the browser and runs them again, aka ABC[restart]ABC[restart]ABC. The hope is that we will not lose too much flake coverage, but will reduce the amount of non-addressable flake that is reported. This also makes it more feasible to implement a timeout-avoiding mechanism for the stability checks; see https://docs.google.com/document/d/1dAlCSHUQldtgWDDTrGJR-ksm19FZZ3k8ppqc5-kSwIk/edit#
stephenmcgruer
force-pushed
the
smcgruer/stability-checks-repeat-onky
branch
from
a1722bd
to
d377806
Compare
stephenmcgruer
changed the title
|
Shall we hack the "verify" logic instead of adding these flags? |
|
It wouldn't be a hack; we could just set the default value for Really depends on how we want developers to use this. With verify, we've moved away from a lot of the defaults already (e.g. no chaos mode on FF), but we've so far done that by setting the flags rather than changing the defaults, which implies that we want any developer running I mean personally I could even see us dropping this flag entirely and just saying you have to |
There was a problem hiding this comment.
Note that bare --verify is currently used in Firefox CI, so changing the meaning of that would be a breaking change for us (technically requiring an RFC &c.). Which isn't to say that we can't do it, but it definitely feels like the default is more likely to find problems, and the flags we're passing are compromises to make the CI work better.
We frequently hear from test authors that they find it frustrating when
the stability checks turn up failures that they cannot reproduce. One
common case comes from the fact that stability checks run the same test
repeatedly without restarting the browser (the AAABBBCCC behavior). This
is unlike any other way we run tests, and can cause some tests to
consistently appear flaky due to global state (e.g. the
origin-isolation/ tests).
To fix this, switch the stability checks to only use 'repeat-restart'
flake detection (previously we used both 'repeat-loop' and
'repeat-restart'). This mode run the tests in entire sets, then restarts
the browser and runs them again, aka ABC[restart]ABC[restart]ABC. The
hope is that we will not lose too much flake coverage, but will reduce
the amount of non-addressable flake that is reported.
This also makes it more feasible to implement a timeout-avoiding
mechanism for the stability checks; see
https://docs.google.com/document/d/1dAlCSHUQldtgWDDTrGJR-ksm19FZZ3k8ppqc5-kSwIk/edit#