Paging @web-platform-tests/wpt-core-team for thoughts. See also #8391, although I don't think we have a clear idea of how slow compared to other stacks and why.

Oh, interesting. When I say Python is too slow for crypto, I usually mean that it's unsuitable for production uses. That may not directly affect WPT, but it has the second-order effect that the Python crypto ecosystem is largely native bindings, so WPT cannot draw on that.

But, yeah, if WPT is already slow even with native bindings for TLS, adding a pure-Python TLS with pure-Python crypto primitives will make things even worse. :-)

I'm marking this priority:roadmap, because I think this is something we need to consider some sort of story for in 2020. The ultimate answer might be that TLS/TLS-adjacent platform features are out of our scope, or it might mean developing a better network stack for wpt, but we need to figure out what that answer is.

I believe @Hexcles mentioned having had historical thoughts on this, which they may already have written up elsewhere.

Switching away from Python for wptserve I think would also impact web-platform-tests/rfcs#23 (cc @louaybassbouss)

We had this conversation at TPAC relating Quic [1].

The summary of that discussion is that whilst people would like to move away from Python for various things (I'd love to rewrite some components in Rust, for example; the manifest update is a clear example of where we've reached the limits of pure-Python performance), there are significant deployment challenges with adding compiled components to the testsuite, particularly if they aren't in C/C++, since browser vendors don't uniformly support the same set of compilers (e.g. Gecko has great Rust support and afaik no support whatsoever for Go). So making this kind of change requires a very compelling story for how to distribute it so it can work on vendor infrastructure.

[1] https://www.w3.org/2019/09/17-testing-minutes.html#item05

Above and beyond @jgraham's comment above:

WPT doesn't have good mechanisms for testing network-related web platform features that sit below the HTTP abstraction. In particular, TLS and TLS-adjacent features are untestable by WPT.

I think we need to have a clear definition of what we expect the scope for WPT to be.

Originally, it didn't include any network layer tests (okay, it relied on HTTP semantics in places, but certainly nothing lower level). Later we added tests for an API that was directly wedded to a protocol (WebSocket) by introducing a server for that. HTTP/2 has a very long and slow history here (needed for testing some things around Fetch), WebTransport and QUIC were raised at TPAC…

From my point of view, it's not clear that it necessarily makes sense to test anything above TCP/UDP all within one repo. As you note, we're very ill-equipped to test anything below the HTTP semantics layer, and to me it's not obvious it necessarily makes sense to include that all here. You need very different infrastructure to break apart the lower layers (and we absolutely should have shared tests there, it's just not clear to me that it makes sense to try and mold that into the infrastructure we already here).

Of course, yes, to some degree certain error conditions from lower layers propagate up (especially with lower level APIs like WebTransport), but the thought of taking on a whole load of extra infrastructure code (given we'd presumably end up with everything above TCP/UDP in this repo) scares me given historically extra infra has been maintained by some sort of vague notion of a WPT infra team (which… doesn't actually exist) and it's not clear those currently in that team have the necessary expertise or time to maintain all that.

Error conditions is not an accurate characterization of the scope here. Random web platform features may cross the divide. For instance, the resource timing APIs need to have defined interaction with 0-RTT, which is where this most recently came up.

If the answer is any web platform feature which interacts with modern features of bits under HTTP is out of scope for WPT, that's a fine answer. It's how we've been operating all this time, treating the WPT aspects of launch processes as yet another translation failure when those processes are applied to network-related features. But this keeps coming up and @foolip requested I file a bug, so here is a bug. Hopefully we can get this written down once and we don't have to repeat this conversation all the time. :-)

FWIW, I'm not opposed to having it all in WPT, we just need to address who will maintain the requisite infrastructure and how we'll get that running across all browser CI systems.

I think we need to have a clear definition of what we expect the scope for WPT to be.

Originally, [...]

Well-said, @gsnedders

I'm not so concerned about scope creep. I think the point of web-platform-tests is to have a shared testsuite for the features that are required to implement a web-compatible browser. There's no expectation that only includes things above the network layer; it's just that those were the lowest hanging fruit when the project started. Indeed one of the key design goals of wptserve was to allow writing tests which interacted with the HTTP layer in ways that wouldn't be possible in normal production servers. As the platform has exposed more of the lower layers of the stack it's only natural that we should have the same requirements there. Punting on the problem and encouraging people to build their own testsuite is bad; third party suites usually can't share infrastructure (e.g. two-way sync, wpt.fyi) and so often end up not providing the same level of ongoing quality assurance that a wpt-intergrated suite might.

That said, I reiterate that whilst I don't think there are philosophical concerns with adding infrastructure to enable tests covering these features, there are more than enough pratical challenges to make up the difference :)

Thanks @jgraham that all aligns with how I see this. In principle I'd be happy for one test suite for everything that can affect web developers, but how to actually test TLS or TLS-adjacent I don't know.