I can reproduce locally

./wpt run --binary `which google-chrome-unstable` --log-mach - chrome origin-policy/content-security/valid-with-semicolon.https.html

only runs two tests and I don't see any console errors in Chrome.

But it does work under ./wpt serve - loading even the non-https version in a browser after serving still shows all 4 subtests.

No further understandings yet; I thought the problem was that 'getmessages' was being sent to the iframe before it registered its listener for it, but that also happens for the working ./wpt serve version.

Could this be related to 644a206#diff-6f05ece846271f4b15119cfe565da64dR72-R96 (omitting the origin policy subdomains from the web driver environment variables)? Although if that's it, then there's probably a separate bug (in testharness.js?) where these tests should be marked as failing instead of passing, if the calls to fetch_tests_from_window don't work.

I don't think its related to that; that should really only affect the webdriver tests.

I've gone very far down the rabbithole here, and I'm not making progress. For some reason, when using wpt run the main frame isn't getting any message_functions messages (e.g. test start, state change, etc) from the iframe. It looks like the tests are running, I just can't sort out the ordering to figure out what is different between the two cases that is causing messages not to happen :(.

One big difference is that wptrunner injects its own setup and config changes, but I don't know how to tell if that is all working as expected or not.

cc @jgraham, in case he has magical knowledge about the workings of RemoteContext/RemoteTest and might know why its working here under wpt serve but not wpt run.

Argh, I had figured it hours ago but had failed to realize that!

The problem is with the testharnessreport.js injected by wptrunner. It re-calls setup on the test environment, passing a filtered list of message_events:

  var props = {output: %(output)d,
               timeout_multiplier: %(timeout_multiplier)s,
               explicit_timeout: %(explicit_timeout)s,
    message_events: ["completion"]};

This causes us to remove our previously-added callbacks for start, test_state, and results, which then means that the iframe doesn't dispatch any messages about them to the main parent window, so the main parent doesn't know about them when reporting results.

The reason I had overlooked it is that if you're using wpt run but just refreshing the test in the browser, you don't get a new version of testharnessreport.js when you refresh (I think). You have to re-run wpt run entirely.

As to a fix, I'm not immediately sure. Adding the other events back into message_events appears to work, but may have unintended consequences.