Transferred this issue to wpt because on the wpt.fyi side we're totally capable of storing all screenshots. The change needs to happen in wptrunner and IIRC for Marionette reftest runner this might require a bit of work as it doesn't return the screen capture if a test passes (I'm not sure). And then there's the question of whether the CI platforms (Azure & Taskcluster) can store the screenshots temporarily (during the run).

We have all the support needed in wptrunner for this, just a question of passing the right flags and storing the right screenshots.

Hmm, I guess my memory was about the default behaviour, which can probably be changed here:

Also IIRC @jgraham previously expressed concerns about performance regressions (especially in Firefox where not all screenshots are dumped to the runner by default).

A few numbers FWIW: so far we have about 68k images in total, less than 1GB for all failing screenshots across all browsers

The support exists here, but we don't have any way to only generate the screenshots we don't have. And in general that seems tricky because we can't hash a screenshot we haven't generated to know if it's been seen before. So having this on by default (even just for GitHub runs) ends up doing a lot of extra work.

If we do a push to a trigger branch with the flags set to generate screenshots always, that should backfill a bunch of data. Could we start by doing that? Later we could consider setting the flag for PRs but not master runs, or doing one run a month with the flag enabled, or only on beta or something.

So your main concern is about performance (specifically, run time), right? We should try and see how much slower it actually is to generate all screenshots.

Tentatively marking backlog, but feel free to upgrade to roadmap @Hexcles if you are planning to work on this anytime soon.