RFC 65: Switching to "Py3-first" for WPT tests on CI #65

ziransun · 2020-10-28T19:09:50Z

No description provided.

rfcs/Py3-first.md

jgraham

Is there a proposal for the date when these changes will occur?

Ms2ger

👍

stephenmcgruer · 2020-10-29T11:28:46Z

Is there a proposal for the date when these changes will occur?

Not yet. I would suggest that we make it part of the RFC to say we aim to switchover shortly after merge, but will announce the switchover in [PLACES] with X days notice.

Before we get there, we need to determine what are the list of blocking issues versus which aren't blocking. The blocking issues I know of are:

The threading issue - this is our riskiest blocker.
The flaky-failure when trying to install Py3 on the Windows test-suite bots.
The other 'switch to Py3' pull requests should probably all be in a reasonably reviewed state before we land the RFC (basically so there are no surprises when it comes to switchover day).

Are there any other blocking issues folks are aware of?

rfcs/Py3-first.md

See the RFC: web-platform-tests/rfcs#65

rfcs/Py3-first.md

Hexcles · 2020-10-29T14:09:00Z

The threading issue - this is our riskiest blocker.

With a tight-ish time frame in front us, I think we should take web-platform-tests/wpt#26111 to work around this for now. We talked about experimenting with multiprocessing; I fully agree this is something we should try, but we probably don't want to block the migration on the experiment.

stephenmcgruer · 2020-10-29T14:10:27Z

With a tight-ish time frame in front us, I think we should take web-platform-tests/wpt#26111 to work around this for now. We talked about experimenting with multiprocessing; I fully agree this is something we should try, but we probably don't want to block the migration on the experiment.

Sorry, last I checked I thought the fix did not actually address the encoding problem. Did the locking added in web-platform-tests/wpt@b7fa472 definitely resolve the encoding import errors?

Hexcles · 2020-10-29T14:12:14Z

Sorry, should've made that clear in the PR. The answer is yes, but see web-platform-tests/wpt#26111 (review) for caveats...

stephenmcgruer · 2020-10-29T14:27:31Z

Sorry, should've made that clear in the PR. The answer is yes, but see web-platform-tests/wpt#26111 (review) for caveats...

I see. That changes my opinion on this; if we have a valid (albeit scary) workaround and are committed to finding a proper path forward that will remove the workaround (which we are; I am happy to own that myself), I don't think the threading issue should block the RFC then.

@jgraham , WDYT about the threading issue given this?

jgraham · 2020-10-29T14:29:57Z

Ideally I'd like us to try the absolute import thing first. If we can't make that work we could fall back to the existing patch, but I think we should be very wary of taking something that pokes at so many internals; if we drop the ball on fixing the underlying issue it seems almost certain we'll see breakage at some point.

stephenmcgruer · 2020-10-29T14:57:47Z

Ideally I'd like us to try the absolute import thing first. If we can't make that work we could fall back to the existing patch, but I think we should be very wary of taking something that pokes at so many internals; if we drop the ball on fixing the underlying issue it seems almost certain we'll see breakage at some point.

Ack, I can own experimenting with that over the next few (business) days.

stephenmcgruer · 2020-11-03T12:44:30Z

Ack, I can own experimenting with that over the next few (business) days.

To keep folks up-to-date on this, I am playing with it in web-platform-tests/wpt#26328 . My next task is to diff a set of trigger runs that I did for Chrome Dev.

stephenmcgruer · 2020-11-09T20:37:49Z

An update:

We believe we have cleared the path for the Windows 3.8 installation issue.
The Python file handler threading issue has a proposed PR by me out for review now, but it's a fairly invasive change and will likely need an RFC of its own anyway.

EDIT: RFC filed for Python file handler change - #68

stephenmcgruer

This depends on #68 (and the related fix), but once that's in this LGTM.

rfcs/Py3-first.md

See the RFC: web-platform-tests/rfcs#65

jgraham

So I generally agree with everything here, but I'd like a Python 3 meta RFC formalizing the timeline for all the steps, rather than just having an RFC for one part of the process and the overall framework in an issue. I can write that.

stephenmcgruer · 2020-11-12T14:48:56Z

So I generally agree with everything here, but I'd like a Python 3 meta RFC formalizing the timeline for all the steps, rather than just having an RFC for one part of the process and the overall framework in an issue. I can write that.

I mean, no objection since you're willing to write it, but isn't that two lines?

November 2020: Py3-first
Feburary 2021: Py3-only

jgraham · 2020-11-12T15:26:52Z

I've written #69 to detail the overall transition plan.

The one thing I realised while writing that that isn't covered here is the behaviour of wpt --py3; I think we need a --py2 option during the transition. I can make a PR for that.

jgraham · 2020-11-12T15:34:28Z

web-platform-tests/wpt#26507

Co-authored-by: Stephen McGruer <stephen.mcgruer@gmail.com>

See the RFC: web-platform-tests/rfcs#65 This affects: * On push: Full test-suite runs. * On pull-request: * The with-pr affected-results check. * The without-pr affected-results check. * The stability checks.

See the RFC: web-platform-tests/rfcs#65

See the RFC: web-platform-tests/rfcs#65 This also affects `infrastructure/ tests: Windows 10`, but that job is currently only manually triggered and is basically unused. Co-authored-by: Mustapha Jaber <mustjab@users.noreply.github.com>

…thon 3 only, a=testonly Automatic update from web-platform-tests [Taskcluster] Switch wpt run on CI to Python 3 only (#26252) See the RFC: web-platform-tests/rfcs#65 This affects: * On push: Full test-suite runs. * On pull-request: * The with-pr affected-results check. * The without-pr affected-results check. * The stability checks. -- wpt-commits: 26c1676d2b6d7632f8de4ec3537896b3f45bd394 wpt-pr: 26252

…afari test-suite runs, a=testonly Automatic update from web-platform-tests [Azure Pipelines] Use Python 3.8.x for Safari test-suite runs (#25044) See the RFC: web-platform-tests/rfcs#65 -- wpt-commits: 21d61bde300f8ff06ea94fdd6eea32e8d2bf2a91 wpt-pr: 25044

…dge test-suite runs, a=testonly Automatic update from web-platform-tests [Azure Pipelines] Use Python 3.8.x for Edge test-suite runs (#24952) See the RFC: web-platform-tests/rfcs#65 This also affects `infrastructure/ tests: Windows 10`, but that job is currently only manually triggered and is basically unused. Co-authored-by: Mustapha Jaber <mustjab@users.noreply.github.com> -- wpt-commits: 79c46570895f9a2bf51b812355b93b2fac116318 wpt-pr: 24952

…thon 3 only, a=testonly Automatic update from web-platform-tests [Taskcluster] Switch wpt run on CI to Python 3 only (#26252) See the RFC: web-platform-tests/rfcs#65 This affects: * On push: Full test-suite runs. * On pull-request: * The with-pr affected-results check. * The without-pr affected-results check. * The stability checks. -- wpt-commits: 26c1676d2b6d7632f8de4ec3537896b3f45bd394 wpt-pr: 26252

…afari test-suite runs, a=testonly Automatic update from web-platform-tests [Azure Pipelines] Use Python 3.8.x for Safari test-suite runs (#25044) See the RFC: web-platform-tests/rfcs#65 -- wpt-commits: 21d61bde300f8ff06ea94fdd6eea32e8d2bf2a91 wpt-pr: 25044

…dge test-suite runs, a=testonly Automatic update from web-platform-tests [Azure Pipelines] Use Python 3.8.x for Edge test-suite runs (#24952) See the RFC: web-platform-tests/rfcs#65 This also affects `infrastructure/ tests: Windows 10`, but that job is currently only manually triggered and is basically unused. Co-authored-by: Mustapha Jaber <mustjab@users.noreply.github.com> -- wpt-commits: 79c46570895f9a2bf51b812355b93b2fac116318 wpt-pr: 24952

…thon 3 only, a=testonly Automatic update from web-platform-tests [Taskcluster] Switch wpt run on CI to Python 3 only (#26252) See the RFC: web-platform-tests/rfcs#65 This affects: * On push: Full test-suite runs. * On pull-request: * The with-pr affected-results check. * The without-pr affected-results check. * The stability checks. -- wpt-commits: 26c1676d2b6d7632f8de4ec3537896b3f45bd394 wpt-pr: 26252 UltraBlame original commit: 199301f4505bc4cadf7abbf46858d03abb01196a

…afari test-suite runs, a=testonly Automatic update from web-platform-tests [Azure Pipelines] Use Python 3.8.x for Safari test-suite runs (#25044) See the RFC: web-platform-tests/rfcs#65 -- wpt-commits: 21d61bde300f8ff06ea94fdd6eea32e8d2bf2a91 wpt-pr: 25044 UltraBlame original commit: 5c0233612758e709865f37f4a3698761f1021638

…dge test-suite runs, a=testonly Automatic update from web-platform-tests [Azure Pipelines] Use Python 3.8.x for Edge test-suite runs (#24952) See the RFC: web-platform-tests/rfcs#65 This also affects `infrastructure/ tests: Windows 10`, but that job is currently only manually triggered and is basically unused. Co-authored-by: Mustapha Jaber <mustjabusers.noreply.github.com> -- wpt-commits: 79c46570895f9a2bf51b812355b93b2fac116318 wpt-pr: 24952 UltraBlame original commit: a27b5f9fc60da817c0a11ce402e816973ccdc831