Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Latest Servo run on wpt.fyi is from March 6 #24257

Closed
jdm opened this issue Jun 19, 2020 · 9 comments
Closed

Latest Servo run on wpt.fyi is from March 6 #24257

jdm opened this issue Jun 19, 2020 · 9 comments

Comments

@jdm
Copy link
Contributor

jdm commented Jun 19, 2020

I don't really know where the code that triggers those runs lives, or where logs are stored from those runs, so I'm filing this here.

@jdm jdm added the wpt.fyi label Jun 19, 2020
@jgraham
Copy link
Contributor

jgraham commented Jun 22, 2020

Seems like everything is running, but the wdspec jobs time out, and we therefore don't ingest the results. I'm not sure what the fix is here. We could increase the number of chunks for wdspec, or change the configuration so servo doesn't run them, or disable tests we don't expect servo to support in that directory.

bors-servo added a commit to servo/servo that referenced this issue Jun 22, 2020
Report exceptions for async script executions to webdriver

Improving the accuracy of the async script execution for our webdriver implementation allows us to run many more webdriver tests without timeouts, which should help with web-platform-tests/wpt#24257.

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27036 and #27035 and #27031.
- [x] There are tests for these changes
bors-servo added a commit to servo/servo that referenced this issue Jun 23, 2020
Report exceptions for async script executions to webdriver

Improving the accuracy of the async script execution for our webdriver implementation allows us to run many more webdriver tests without timeouts, which should help with web-platform-tests/wpt#24257.

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27036 and #27035 and #27031.
- [x] There are tests for these changes
bors-servo added a commit to servo/servo that referenced this issue Jun 23, 2020
Report exceptions for async script executions to webdriver

Improving the accuracy of the async script execution for our webdriver implementation allows us to run many more webdriver tests without timeouts, which should help with web-platform-tests/wpt#24257.

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27036 and #27035 and #27031.
- [x] There are tests for these changes
bors-servo added a commit to servo/servo that referenced this issue Jun 23, 2020
Report exceptions for async script executions to webdriver

Improving the accuracy of the async script execution for our webdriver implementation allows us to run many more webdriver tests without timeouts, which should help with web-platform-tests/wpt#24257.

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27036 and #27035 and #27031.
- [x] There are tests for these changes
@jdm
Copy link
Contributor Author

jdm commented Jun 25, 2020

Ah, we worked around that locally by only running with a fixed set of supported test-types.

@jdm
Copy link
Contributor Author

jdm commented Jun 26, 2020

That being said, it's not clear to me why Safari and Edge are not affected by that problem, because I don't see any support in their executors for the new print-reftest test type.

@jdm
Copy link
Contributor Author

jdm commented Jun 26, 2020

Looks like webkitgtk is actually hitting the same issue:

 0:14.49 INFO Running print-reftest tests
 0:14.50 ERROR Unsupported test type print-reftest for product webkitgtk_minibrowser
 0:14.50 INFO Got 0 unexpected results
 0:14.50 SUITE_END

@jdm
Copy link
Contributor Author

jdm commented Jun 26, 2020

Ok, so there are only explicit print-reftest jobs for 1) servo, 2) firefox, 3) chrome, 4) webkitgtk.

@stephenmcgruer
Copy link
Contributor

Safari and Edge run on Azure Pipelines (so we can access Windows and MacOS). They use a totally different way of choosing their runs and uploading the results, see .azure-pipelines.yml and the results_edge_dev/etc jobs.

In short they run:

python ./wpt run --yes --no-manifest-update --no-restart-on-unexpected --no-fail-on-unexpected --install-fonts --this-chunk $(System.JobPositionInPhase) --total-chunks $(System.TotalJobsInPhase) --chunk-type hash --log-wptreport $(Build.ArtifactStagingDirectory)/wpt_report_$(System.JobPositionInPhase).json --log-wptscreenshot $(Build.ArtifactStagingDirectory)/wpt_screenshot_$(System.JobPositionInPhase).txt --log-mach - --log-mach-level info --channel dev edgechromium

Then they manually upload their results to wpt.fyi, via tools/ci/azure/fyi_hook.yml.

I suspect for them, the print-reftests will fail to run but that won't stop runs being uploaded.

@jgraham
Copy link
Contributor

jgraham commented Jun 26, 2020

We should probably change the taskcluster yml to only run print reftests for browsers that support them. Unfortunately that's quite a lot of duplication right now. We could add more logic to make it easier or just live with a big copy&paste job

@stephenmcgruer
Copy link
Contributor

Is anyone owning the next steps on this issue? I'm downgrading from urgent to roadmap as it has no clear owner, has had no activity for multiple weeks, and we've survived since March without Servo runs. (Which is not to say that's ok, but to acknowledge that its difficult to consider this 'urgent' if 4 months can pass with the issue existing).

clopez added a commit to clopez/wpt that referenced this issue Jul 13, 2020
* print-reftests are currently not supported for this two browsers.
So we need to avoid triggering those runs on taskcluster for them to
avoid failing the whole run. This fixes: web-platform-tests#24360 and web-platform-tests#24257.
* This patch removes the print-reftest suite from the general block
and adds a new block for it and for the browsers that support this
tests on taskcluster (chrome and firefox).
clopez added a commit to clopez/wpt that referenced this issue Jul 13, 2020
…#24576)

* print-reftests are currently not supported for this two browsers.
So we need to avoid triggering those runs on taskcluster for them to
avoid failing the whole run. This fixes: web-platform-tests#24360 and web-platform-tests#24257.
* This patch removes the print-reftest suite from the general block
and adds a new block for it and for the browsers that support this
tests on taskcluster (Chrome and Firefox).
clopez added a commit that referenced this issue Jul 13, 2020
* print-reftests are currently not supported for this two browsers.
So we need to avoid triggering those runs on taskcluster for them to
avoid failing the whole run. This fixes: #24360 and #24257.
* This patch removes the print-reftest suite from the general block
and adds a new block for it and for the browsers that support this
tests on taskcluster (Chrome and Firefox).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants