Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chrome/Firefox beta runs aren't reliable #19206

Closed
foolip opened this issue Sep 22, 2019 · 7 comments
Closed

Chrome/Firefox beta runs aren't reliable #19206

foolip opened this issue Sep 22, 2019 · 7 comments
Assignees
@foolip
Labels
infra priority:roadmap Taskcluster

Comments

@foolip
Copy link
Member

foolip commented Sep 22, 2019

#18608 was causing Chrome and Firefox beta runs to be missing, but last week's run still didn't work:
https://tools.taskcluster.net/groups/c7HqBfGPRQil9QX7cuPPAA

One task failed, and so no results at all were uploaded to wpt.fyi. The failure cause was just a network error for an external dependency.

@jgraham between trying to reduce external dependencies and triggering beta runs daily, which would you prefer?

I ask because I've been trying for a while now to look at stable→beta for regressions, but when it fails one has to wait a week for the next attempt.
@foolip
Copy link
Member Author

foolip commented Sep 22, 2019

I tried to get results to align with the latest stable run, but that failed:
42b4a3fa60#commitcomment-35182287

@jgraham I'll file an issue about it.

foolip added a commit that referenced this issue Sep 22, 2019
@foolip
[Taskcluster] run Chrome and Firefox beta daily
eeb9229 
To mitigate #19206.
@foolip foolip mentioned this issue Sep 22, 2019
Closed
@foolip
Copy link
Member Author

foolip commented Sep 22, 2019

The issue I filed is #19207.

Proposed #19212 to mitigate the lack of beta runs.

@foolip
Copy link
Member Author

foolip commented Sep 23, 2019

And as luck would have it the weekly beta run was successful without human intervention for the first time since August 12:
https://wpt.fyi/results/?sha=ba8a54c742&label=beta&label=master&max-count=1

@foolip
Copy link
Member Author

foolip commented Sep 23, 2019

All beta runs so far: https://wpt.fyi/runs?label=master&label=beta&max-count=100

@foolip
Copy link
Member Author

foolip commented Sep 23, 2019

Example of the sort of view that's worthwhile to look at:

@lukebjerring note the use of status:!missing to exclude some noise due to unstable subtest names. Maybe this should be included in the /insights queries?

@foolip foolip mentioned this issue Sep 23, 2019
Open
@foolip
Copy link
Member Author

foolip commented Sep 23, 2019

I've closed #19212 since it's not so clear what the current reliability is, so #14210 to understand the situation should be higher priority.

@foolip foolip self-assigned this Sep 24, 2019
@foolip
Copy link
Member Author

foolip commented Oct 8, 2019

We've had successful beta runs for the past 3 weeks now:
https://wpt.fyi/runs?label=master&label=beta

@foolip foolip closed this as completed Oct 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@foolip foolip

Labels
infra priority:roadmap Taskcluster
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@foolip