This basically depends on whether we want to use subtests to describe individual independent things / testcases on machines that don't necessarily depend on each other (A), or just use them as a way to group / give a heading to a bunch of statements (B).

The current behaviour (both in python and perl) is to add the an error message to failed_tests, but keep going with the next one if I'm not mistaken (cc @globin #74089), which assumes (A)

However, I think we mostly do B in many tests, even though we probably shouldn't. WDYT?

I don't think we should change semantics here with the perl driver still around.

This basically depends on whether we want to use subtests to describe individual independent things / testcases on machines that don't necessarily depend on each other (A), or just use them as a way to group / give a heading to a bunch of statements (B).

It generally makes sense to enable the user to decide which things should make tests fail and which should not.

The current behaviour (both in python and perl) is to add the an error message to failed_tests, but keep going with the next one if I'm not mistaken (cc @globin #74089), which assumes (A)

True, but i observed this behavior to do no good in the many many tests that i have ported.

However, I think we mostly do B in many tests, even though we probably shouldn't. WDYT?

The vast majority of tests that i have seen assumes behavior B. Behavior B is also a good default setting in my opinion.

We could longterm give users some wrapper that is similar to subtest but that is explicit in isolating exceptions from the code outside so users can kind of add an error summary.
Such summary as "the following subtests failed" is not helpful in many tests, because the failing subtests were collected, but then the whole test times out after some steps did not work that would have been prerequisites to following steps.

I agree that a lot of tests currenly assume B.

I'm just hesitant introducing semantic changes with both python and perl driver around, so would really feel better if we pick this up when the perl driver isn't around anymore.

I'm just hesitant introducing semantic changes with both python and perl driver around, so would really feel better if we pick this up when the perl driver isn't around anymore.

I agree that it is not cool to have different semantics in different implementations that share the same names.

But we have a lot of things in subtests that are necessary prerequisites to their next steps, but won't blow up visibly, leading to tests that time out (e.g. something must be done, but that fails, and then a test loops on wait_until_x which then can never happen) and are hard to debug.

Regarding the minority of still existing perl tests that remain untouched and working until they are ported (and people will see that they get more fails instead of less then, which is always better than the other way around), i suggest putting this in.

Just want to mention if there's any decision on this please merge after branch off.

I stumbled more than once into python tests failing in a previous subtests, while I was staring at the later (still running) output.

As this doesn't change semantics of passing tests, and as tests run isolated, I don't have any strong objections against this.

I pushed some cleanups, PTAL.

@flokli looks good!