@burg @stephenmcgruer I expect this won't be possible to land before #21628, making the problems with Catalina somewhat urgent, if it means we can't have any new results for Safari STP until it's resolved.

Thanks Philip, I had not realized that STP 109 was Catalina-and-up only. That certainly raises the priority; as a note the tracking issue for the problem is actually #21751, not #21628 (the latter is the pull request where we have been debugging).

cc @othermaciej as an FYI that if we are unable to resolve the Catalina issue here, we will no longer be able to get WPT results on new Safari Tech Preview versions.

@mustjab - is there someone from the Azure Pipelines team we could talk to about this issue? As per #21751 (comment), no-one has been able to reproduce this issue on anything outside of the Azure Pipeline images, and notably @burg found when looking at the sysdiagnose dump that "no error is returned on the file write operation" - but the plist file still isn't written. This all makes it seem like there's something special about the Azure Pipelines setup that we're failing to account for in our attempts to reproduce.

In the meantime, I'm going to try both the Catalina upgrade + STP 109 at the same time, just in case something new and wonderful has happened and the problem has gone away...

If there's a suspicion that this is something about Azure Pipelines, then it would be worth trying to run a few tests on other CI systems, like GitHub Actions, which doesn't appear to use the same underlying infra for macOS. We might not have enough capacity there to actually use it for our runs, but it might help form a better hypothesis of the problem.

@stephenmcgruer Sent mail to Azure Pipelines MacOS team, will let you know when I hear back.

I just tried my own suggestion in foolip/safari-ci-starter-kit@ecaf066 and it seems to me the problem is not specific to Azure Pipelines. Using macOS Catalina in all of Azure Pipelines, CircleCI, Cirrus CI, GitHub Actions, Travis CI, all of them fail in the same way: "Could not create a session: You must enable the 'Allow Remote Automation' option in Safari's Develop menu to control Safari via WebDriver"

Is there something that everyone has to do with macOS in a CI environment that isn't the case on a developer machine? What's going on? :D

Is there a command to test for that difference?

Is there a command to test for that difference?

To literally check for SSH we could look for SSH_TTY, SSH_CONNECTION, etc.

For other approaches, I'm not sure. Seems like we would be better off asking the Azure Pipelines team how the sessions work?

Obsoleted by #24631 (STP 110)