@jugglinmike we've already decided to improve the visibility of the the deployment in Q4 which will involve touching this job, so assigning to you.

If it keeps failing like this, since it's not possible to discover where it's been deployed to even when successful, then I'd like to disable it until that work starts.

It appears that the permissions model of the GitHub Actions feature has changed and that the new implementation is fundamentally incompatible with the solution we designed. I believe we'll need to re-implement the "git tag" and "Pull Request label" operations on an external server.

Comparing the pull requests which have received the pull-request-has-preview label against the pull requests which have not received the label, the Action only seems to succeed when running for a pull request which has been opened in this repository. In other words: it doesn't work for forks. This is an unacceptable limitation for a feature intended to support all contributors generally.

When initially implemented, we demonstrated that this feature would work for contributors that do not have "write" access to the repository. About three weeks ago, a GitHub employee wrote:

For the recent update for GitHub Actions we have made a lot of changes to how workflows are run.

The documentation referenced in our script no longer exists, and the new documentation states:

The permissions for the GITHUB_TOKEN in forked repositories is read-only. For more information about the GITHUB_TOKEN, see "Virtual environments for GitHub Actions."

Which leads me to believe that we'll need to grant the necessary permissions to an external service and re-write the GitHub Action to notify that service.

This is a non-trivial amount of work, though, so I'd appreciate it if @foolip or any of the other infrastructure maintainers could verify my interpretation!

A token with read-only access to the target repository makes good sense for PRs from forks, and I can understand why GitHub changes it to work like that.

Without write access we could still deploy (by polling open pull requests) but can't in any way point to it from the pull request. I agree it seems like we need to have an external service doing the commenting/tagging/whatever.

Pending a new solution I've sent #19456, since it likely won't remain in that form and it's not working as intended.

Whether we post comments or use GitHub's deployments API I'm inclined to say that the same code that monitors for open PRs in need of deployment would also post back when the deployment is done. That there are multiple instances for load balancing probably complicates this somewhat, but it has to be possible to answer the question "will the URL load" somehow.

GitHub Actions supports triggering Workflow events on a pre-defined schedule. A quick experiment on the wpt-actions-test project proved that the token supplied to these executions has write permissions.

By querying the GitHub API, we could implement issue labeling and commit tagging completely in-repository. From the perspective of maintenance and simplicity, this seems preferable to the solution I suggested above. Some risks come to mind, though.

1. Noise. We'll be creating a lot of no-op invocations to the project's Actions log. The UI supports filtering by Workflow title, so if anyone needs to review the history, they can easily ignore the extra information.
2. API rate limiting. Probably we won't need to issue too many requests to achieve this, but it's worth considering. We could mitigate the risk by only considering pull requests that have had activity in the past 24 hours (via the updated: search syntax) and those that have been submitted directly to WPT (filtering out bots using the -author: search syntax).
3. Responsiveness. We've designed wpt.live to manage pull request deployments by polling the git repo for tags. If we define those tags through a polling approach, we'll have layered two polling strategies.

Rather than mirroring pull requests from forks automatically, perhaps we can trigger it by a special label being added and reacting to that event? Can you check if the token for those actions has write access?

The documentation for the "Pull Request Event" in GitHub Actions has the following note about "pull request" Events for forked repositories:

The permissions for the GITHUB_TOKEN in forked repositories is read-only. For more information about the GITHUB_TOKEN, see "Virtual environments for GitHub Actions."

The referenced document shows that "Access by forked repos" is read-only for all available Events.

Alright, I guess that does make sense, it'd be a more complicated model to reason about if what I suggested did work.

Since the workflow no longer exists I'm going to close this issue. @jugglinmike can you file another, if there isn't one already, for getting deployments to wptpr.live working and reported on PRs? We can discuss there what the tradeoffs are, and I'd like to get @Hexcles's feedback there as well.

Sure! See gh-19607.