I confirmed locally that no_proxy='*' ./wpt run --yes --no-manifest-update --manifest MANIFEST.json --metadata infrastructure/metadata/ --log-tbpl - --log-tbpl-level info --webdriver-arg=--diagnose --channel preview safari infrastructure/ produces a lot of logs in ~/Library/Logs/com.apple.WebDriver/.

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27221 showed this to work for the infrastructure/ tests. There were 10 log files produced:

safaridriver.1705.oakdiu.txt
safaridriver.1720.fmgfke.txt
safaridriver.1724.edimyy.txt
safaridriver.1729.vnyfkp.txt
safaridriver.1731.mtllhj.txt
safaridriver.1741.hquxlw.txt
safaridriver.1745.zaxjwp.txt
safaridriver.1749.dasyrk.txt
safaridriver.1753.mlkuzq.txt
safaridriver.1759.cxlygb.txt

The first was the largest of them, and I've pasted it into https://gist.github.com/foolip/5bca5e1504bc9206e0763b114d8e798f.

@smfr @youennf @burg can any of you review this and say if it's adding valuable information?

Once #18397 has been resolved I can trigger full runs and see if there's something interesting in the logs.

Full run of the two Safaris in progress to test this: https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27258

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27258 finished and has the logs as artifacts. safaridriver-logs.zip is over 450 MB, so always collecting these logs might not be a great idea.

There are 155 log files, and correlating them to a failed job isn't going to be easy since they're named by process ID and that isn't included in the main logs, right @gsnedders?

@smfr @youennf @burg can any of you review this and say if it's adding valuable information?

At the end of the log I see:
2019/08/15 07:33:21:339 AutomationProtocol: RECV {"error":{"code":-32000,"message":"JavaScriptTimeout;","data":[{"code":-32000,"message":"JavaScriptTimeout;"}]},"id":425} 2019/08/15 07:33:21:340 HTTPServer: --> HTTP 408 2019/08/15 07:33:21:340 HTTPServer: --> {"value":{"error":"timeout","message":"","stacktrace":""}}

@foolip, these logs are pretty much required to investigate any safaridriver failures. That said, I think they are too big to always save, and hard to sort out when you have a handful of them.

The --diagnose option is designed to handle situations where safaridriver crashes immediately or in response to New Session command failing. The 'safari:diagnose" capability is more fine-grained and creates log files with the session UUID in it. This may be better for your purposes if you plan to re-run failing tests with additional diagnostics enabled.

@burg since this will be used for debugging problems, we could set it up so that logs are only collected when system.debug is set in Azure Pipelines, or something like that. Let's see if we can use the logs to get to the bottom of #16229 before trying to land this, since we won't want to enable it by default we'll need to start debug runs and then we can just as well start them from a branch.

@youennf I've invited you and @burg to the Azure Pipelines project in the hope that you'll be able to trigger runs yourselves. I'd also be happy to start runs if you can't make it work.

I'm going to close this as it hasn't been necessary for debugging. @burg if you think having this for all runs would be valuable, or a way to enable it for select runs, please let me know.

@burg yeah it would be easy enough to keep this configuration in the repo and ready to be triggered. Have you been using the triggers/safari_preview branch before, or triggering runs from the Azure Pipelines console? Depending on how you do it I'd need to put the control knob in a different place.

If you'd be willing to make changes to a branch to trigger this, then I could just add a variable that defaults to false that could could set to true in one of the YAML files.

I've done this in #21367 now.