Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Manifest download always fails on Azure Pipelines, leading to full rebuild #18928

Open
foolip opened this issue Sep 9, 2019 · 4 comments
Open

Manifest download always fails on Azure Pipelines, leading to full rebuild #18928

foolip opened this issue Sep 9, 2019 · 4 comments
Labels
Azure Pipelines infra priority:backlog

Comments

@foolip
Copy link
Member

foolip commented Sep 9, 2019

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=30178 or probably any other build on Azure Pipelines shows something like this in the "Update manifest" step:

WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18746 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18835 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18922 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18923 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18892 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18882 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18868 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18908 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18918 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18916 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18328 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18865 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18715 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18845 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18842 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18900 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18901 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18880 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18877 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18899 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18856 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18895 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18879 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18893 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18770 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18896 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18891 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18890 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18889 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18884 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18873 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18876 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18894 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18881 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18843 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18885 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18857 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18875 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18837 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18406 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18641 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_18863 failed
WARNING:manifest:Fetching https://api.github.com/repos/web-platform-tests/wpt/releases/tags/merge_pr_14604 failed
WARNING:manifest:No generated manifest found

This is probably because we have to GitHub token and the machines are exceeding the GitHub API quotas. This adds about 3 minutes to every job.

@thejohnjansen @mustjab I've tried to find if there's a token provided by Azure Pipelines that we could use, but I can't find anything. The service connection has a token, but can it be exposed to the steps to use?

Note that GitHub Actions has this capability, where GITHUB_TOKEN is always in the secrets store and can be used in any steps using template syntax.
@mustjab
Copy link
Contributor

mustjab commented Sep 9, 2019

@foolip Checked with Azure Pipeline team and they unfortunately don't expose GitHub tokens.

@foolip
Copy link
Member Author

foolip commented Sep 9, 2019

Hmm, I guess as a workaround we could use a token added to the secrets store manually.

@foolip
Copy link
Member Author

foolip commented Sep 9, 2019

@zcorpan and I were discussing using a personal access token for @wpt-pr-bot for this, but then that seems risky. If there's a quota and that applies per-account and not per-token, then having Azure Pipelines exhaust the quota could cause other things that wpt-pr-bot does to fail. That'd be very bad.

The most hygienic would be to have a dedicated account with no access to anything, and only one token. I'll see if I can wire up the token with my own account first, and if it works register an account just for this.

@foolip
Copy link
Member Author

foolip commented Sep 9, 2019

@jgraham I'm curious if Taskcluster has a GitHub token available, or if the same problem happens there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Azure Pipelines infra priority:backlog
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@foolip @mustjab