Here's an overview of how wptrunner deps work on the Chromium side:

• we are manually rolling tools (including a subset of tools/third_party) on a ~regular basis. During this process we may notice missing/stale deps and update appropriately (by also rolling another subdir of tools/third_party, or installing the dep)
• we also have installed dependencies with pinned versions available for our infra. They don't come directly from pip but our own package server. The consequence of this is that it does take some work to get a new (or updated) package onto this server. This is currently a manual process.
• when a particular dependency is both vendored and installed, our system prefers the installed one.

Chromium's use of wptrunner has been broken by new dependencies in the past, but it happens infrequently. We are almost always "behind" on versions. That is, we don't react to every single minor version update of deps. Instead, we roll up to the latest version of a package when something breaks (we get a chance to intervene without breaking the CI because we roll tools manually). Getting up to date is usually an easy process, although it can be slow due to the manual work involved to get a new package onto the package server. Transitive dependencies have been an issue (ie: you install one package but then realize you need a bunch more) -- this happened only once, though, as part of the py3 upgrade (which was a major change).

Thanks @LukeZielinski, that's super helpful!

Given "our own package server ... currently a manual process" would it be fair to say that it'd be an improvement if dependencies in WPT were updated rarely, on a very predictable schedule like quarterly? Right now we're landing pyup changes as they come in.

And am I right to assume that moving to only installed dependencies would be better, to not have to worry about vendored+installed compatibility and which takes precedence? (It seems to me that in WPT, the vendored deps will take precedence. Oops?)

web-platform-tests/wpt#24829 has some good discussion too.

On the gecko side:

• We update the tools with the rest of the changes; there's no special handling
• When stuff breaks on import it usually requires emergency patches. But I don't recall that being due to dependencies in particular (it's more often stuff like "wptrunner is broken in some configuration that we don't run on GitHub").
• We don't really use the installed dependencies for anything in CI (they can be used by things that run locally). Indeed, to a reasonable approximation, we can't use PyPI dependencies in CI because we're using a mirror that requires manually updating packages and the process for that is laborious. There is talk about making it easier but there's no specific timeframe for that.
• We are also a bit special because the mozbase and marionette packages canonically live in m-c and so we get the latest version of those rather than whatever's on PyPI. That's more normally a problem the other way around.

Basically for Gecko, at this time, moving toward more installed deps would be A Bad Thing, and in general we don't have loads of problems with the current setup where actually running tests can be done using entirely in-tree deps (at least when the tree is mozilla-central).

Thanks @LukeZielinski, that's super helpful!

Given "our own package server ... currently a manual process" would it be fair to say that it'd be an improvement if dependencies in WPT were updated rarely, on a very predictable schedule like quarterly? Right now we're landing pyup changes as they come in.

I'd consider this an improvement yes. Mostly in terms of planning, so you could dedicate some time each quarter (or whatever) to get your deps figured out, and not have to scramble.

And am I right to assume that moving to only installed dependencies would be better, to not have to worry about vendored+installed compatibility and which takes precedence? (It seems to me that in WPT, the vendored deps will take precedence. Oops?)

All-vendored or all-installed both seem better than the hybrid thing that exists today. One argument I could see against all-vendored deps is that it checks a whole pile of random code into the tree (pollutes codesearch results for instance). Although we've never tried to go down this path, so there could be technical challenges I'm not aware of.

Thanks @jgraham, that's very helpful!

In case you saw it, I want to be clear that the all-installed web-platform-tests/wpt#28796 isn't really the direction I'm hoping for here, that's was an exploration to figure out what the dependencies actually are.

It sounds like a requirements from Gecko is that it should be easy import WPT and all of its dependencies so that everything needed is in-tree, and no network access is needed while running WPT in Gecko. Having precisely the dependencies that Gecko doesn't already have manually vendored into WPT meets that requirement.

A rough sketch of the direction I am hoping for now:

• all direct and indirect dependencies tracked, with versions and metadata like "needed for ./wpt manifest" at some granularity
• Gecko, Chromium and WebKit can easily compute the subset of the dependencies they need, and either put them into their own package servers, put them into the tree, or something else
• We could keep a tools/third_party/ in-tree which is exactly what Gecko needs, if it comes to that
• ./wpt run still just does the right thing for users (and maybe ./wpt install can also install all needed dependencies for later ./wpt run --i-am-offline)

So one reasonable goal here would be to remove localpaths.py. I wonder if something like the following could work:

• We have a requirements_vendored.txt file that installs everything under third_party into the virtualenv by path.
• We additionally have a requirements_thirdparty.txt that lists the specific versions of those depencies that we're using
• We have a job that fails in CI if the vendored version doesn't match the specified version

That would have the following properties:

• dependabot or pyupbot could be used to alert us when something in third_party is updated.
• It would be possible for users to use either the vendored dep or the in-tree one at their preference, knowing that both are the same version (this does change if we have local patches; idk if that's a current problem).

Getting rid of localpaths.py is indeed an appealing part of this.

It would be possible for users to use either the vendored dep or the in-tree one at their preference, knowing that both are the same version

Yes!

this does change if we have local patches; idk if that's a current problem

I looked at the recent commits of everything in web-platform-tests/wpt#28801 and didn't find anything with local modifications. Only pywebsocket3 is problematic because it's not yet available on pypi, see GoogleChromeLabs/pywebsocket3#30.

One concern here is startup latency; if every wpt run call is invoking pip and checking that all the packages are installed it's going to add noticable startup time.

For WebKit, there's a few notable cases:

1. Running LayoutTests, which in the future should require wpt.manifest
2. Running check-style, which in the future should require wpt.lint
3. Running WebDriverTests, which requires wptrunner etc. (though I think safaridriver might not use wptrunner, merely using pytest directly?)

Historically, the goal was for at least ./wpt manifest and ./wpt lint to be runnable without any external dependencies (though I'm not sure how much value there is in the former?), but we seem to have gone beyond that (e.g., with hyper and h2). One of the problems with the status-quo is I think it's unclear why things are vendored v. put in as a requirement?

At least from my point-of-view, I'd rather we avoid the status-quo where we have our tools adding our own versions of the packages to sys.path, rather than how other tools (e.g., pip) do vendoring whereby they end up with imports like from pip._vendored import html5lib. That is probably the most essential thing to do.

To note, I think pytest might be the only thing we vendor which vaguely regularly has any API breakage? Maybe h2 too?

All-vendored or all-installed both seem better than the hybrid thing that exists today. One argument I could see against all-vendored deps is that it checks a whole pile of random code into the tree (pollutes codesearch results for instance). Although we've never tried to go down this path, so there could be technical challenges I'm not aware of.

One problem is that some of the packages that wptrunner requires depend on compiled extensions (such as Pillow). We need to install them somewhere as a result.

One concern here is startup latency; if every wpt run call is invoking pip and checking that all the packages are installed it's going to add noticable startup time.

That's precisely one of the things I'd like to resolve here. When I run ./wpt run myself, I can tell that pip is being called at least twice based on the deprecation messages. This doesn't happen in vendor CI, of course, but I'd like to get a better faster experience in the CLI too :)