Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test video resizing during playback #19030

Merged
merged 1 commit into from Nov 28, 2019
Merged

Conversation

foolip
Copy link
Member

@foolip foolip commented Sep 12, 2019

Fixes #17821.

New media files are from that issue.

@foolip
Copy link
Member Author

foolip commented Sep 13, 2019

@foolip
Copy link
Member Author

foolip commented Sep 13, 2019

Still timeout in Safari. I tried playing the file locally and the video is corrupted. Added 'ended' to the event list to fail if it's fired.

@foolip
Copy link
Member Author

foolip commented Sep 13, 2019

Good, test results are what I wanted now.

@eric-carlson @jernoble FYI that this could be an incoming Safari-specific failures. If it's a problem with the media file and Safari should support this sort of this, that'd be great to know.

@guest271314
Copy link
Contributor

Instead of using static files one variation of the test is to use the code which produces the file at each browser with MediaRecorder set the src to an object URL then run the test. Perhaps that might be a test for MediaRecorder and HTML <video> element.

@guest271314
Copy link
Contributor

Does Safari support MediaRecorder?

@foolip
Copy link
Member Author

foolip commented Sep 13, 2019

I dunno, I'd prefer to separate the two, this test is revealing a lack of interop on this specific input that using MediaRecorder might mask.

@guest271314
Copy link
Contributor

Have not compiled a full list, though there are certainly more than one interoperability issues re MediaRecorder, MediaStream and MediaStreamTrack implementations between Firefox and Chrome.

@guest271314
Copy link
Contributor

This issue describes some of the interoperability issues re the previous comment w3c/mediacapture-fromelement#78.

@guest271314
Copy link
Contributor

@foolip This demonstrates that the test can fail at Chromium, Chrome when a WebM file produced by Chromium, Chrome implementation of MediaRecorder is used https://plnkr.co/edit/AovnkZ?p=preview.

aarongable pushed a commit to chromium/chromium that referenced this pull request Oct 1, 2019
NOTE: this CL replaces, https://chromium-review.googlesource.com/c/chromium/src/+/1828226
which was submitted in error.

This changes the behavior implicit resizing of for VP8/9 video. By
implicit, we mean streams where the dimensions change without a new
MSE init segment that explicitly describes the new size.

Prior to this CL, we took whatever resolution was specified by the
container metadata and scaled any later differing resolution into an
element of that original size. Many of our decoders worked this way
before https://chromium-review.googlesource.com/c/chromium/src/+/1026992/,
which shifted the consensus is to allow dynamic resize. This CL makes
the libVPX wrapper consistent with those.

When the size changes we will update videoWidth and videoHeight and
fire a 'resize' event.

Bug: 992235,837337
Test: web-platform-tests/wpt#19030, with --disable-features=FFmpegDecodeOpaqueVP8

Change-Id: I7c65810c545511cfa3441c7cbb9e2771159bc88a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1832478
Commit-Queue: Chrome Cunningham <chcunningham@chromium.org>
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Cr-Commit-Position: refs/heads/master@{#701794}
@chcunningham
Copy link
Contributor

Conceptually this test LGTM. I've updated chrome to ensure dynamic framesizes are supported for all vp8 decoders. Verified this test still passes when libvpx is used.

Re: media files, at a glance the webm file looks sane. I'm not savvy enough to validate the mp4.

@zcorpan
Copy link
Member

zcorpan commented Nov 8, 2019

The test LGTM. I haven't reviewed the media files themselves.

The expected events are in this order:

  1. resize https://html.spec.whatwg.org/multipage/media.html#loading-the-media-resource:event-media-resize
  2. playing https://html.spec.whatwg.org/multipage/media.html#ready-states:notify-about-playing
  3. resize https://html.spec.whatwg.org/multipage/media.html#the-video-element:event-media-resize
  4. probably the test finishes before ended event.

(I recall a Presto test for this from an eon ago, nice to see this finally come to wpt!)

@foolip
Copy link
Member Author

foolip commented Nov 8, 2019

wpt-chrome-dev-stability is failing on Taskcluster like this:

Unstable results

Test Subtest Results Messages
/html/semantics/embedded-content/the-video-element/resize-during-playback.html mp4 video FAIL: 2/10, PASS: 8/10 assert_equals: Expected resize event, but got playing event instead expected "resize" but got "playing"
/html/semantics/embedded-content/the-video-element/resize-during-playback.html webm video FAIL: 1/10, PASS: 9/10 assert_equals: Expected resize event, but got playing event instead expected "resize" but got "playing"

My interpretation is that the event order isn't reliably in the expected order, but I think this is a case where flaky fail is the best case.

Still, I'll rebase to see if the problem has gone away with a newer version of Chrome. Not reproducing isn't evidence that it's really fixed, of course.

foolip added a commit that referenced this pull request Nov 14, 2019
@foolip foolip force-pushed the foolip/resize-during-playback branch from 03bcb6d to ec9cc20 Compare November 14, 2019 10:59
@guest271314
Copy link
Contributor

guest271314 commented Nov 14, 2019

@foolip

1s_red_400x300_1s_green_200x150.mp4.zip
(Does not resize at all media players, use the file at #19030 (comment))

@guest271314
Copy link
Contributor

@fooip Can you confirm the .mp4 video resizes at Safari?

foolip added a commit that referenced this pull request Nov 15, 2019
@foolip foolip force-pushed the foolip/resize-during-playback branch from ec9cc20 to d36965b Compare November 15, 2019 18:03
foolip added a commit that referenced this pull request Nov 15, 2019
@foolip foolip force-pushed the foolip/resize-during-playback branch from d36965b to cd341f3 Compare November 15, 2019 18:06
@foolip
Copy link
Member Author

foolip commented Nov 15, 2019

@guest271314 are you entirely sure the mp4 file does correctly resize half way through? Is there a browser or a player where it does resize? I ask because it doesn't seem to resize in Firefox, but WebM does in Firefox.

@guest271314
Copy link
Contributor

@foolip No. Am at a different OS. Erroneously just used ffmpeg without appropriate patches https://bugs.chromium.org/p/webm/issues/detail?id=1642#c17 then uploaded the MP4 file here. Today have been working on compiling FFmpeg from the linked branch, applying patches, then upload here file, again, here.

@guest271314
Copy link
Contributor

guest271314 commented Nov 15, 2019

@foolip
1s_red_400x300_1s_green_200x150.mp4.zip

Tested at Chromium 80 and Firefox 69.

Was not able to build FFmpeg and apply the patches, again, yet.

The procedure that used was 1) Use <canvas> to draw frames with mimeType set to "video/x-matroska;codecs=h264" (interestingly when canvas.captureStream() to draw frames is not used video does not resize even with H264 codec set https://plnkr.co/edit/uCnx1R?p=info); 2) Copy the H264 encoded video from Matroska to MPEG container, without re-encoding $ /usr/bin/ffmpeg -i 1s_red_400x300_1s_green_200x150.mkv -c:v copy 1s_red_400x300_1s_green_200x150.mp4.

Fixes #17821.

New media files are from @guest271314 in the review.
@foolip foolip force-pushed the foolip/resize-during-playback branch from ea041ce to d42eea3 Compare November 27, 2019 13:16
@foolip
Copy link
Member Author

foolip commented Nov 27, 2019

@guest271314 I've applied the PR you sent against this branch and rebased it, let's see what happens now.

@guest271314
Copy link
Contributor

@foolip Do not currently have access to Safari to test. Tried both WebM and MP4 videos at Epiphany (3.28.5) to simulate Safari. Neither video resized. Tried to install Epiphany Technology Preview got error

$ flatpak install https://nightly.gnome.org/repo/appstream/org.gnome.Epiphany.Devel.flatpakref

error: No such ref 'app/org.gnome.Epiphany.Devel/i386/master' in remote gnome-nightly

Therefore currently do not have a means to verify Safari output, which is currently for MP4

FAIL message: assert_equals: Expected resize event, but got ended event instead expected "resize" but got "ended"

and for WebM

PRECONDITION_FAILED message: webm supported

Am not sure what the message

PRECONDITION_FAILED message: mp4 supported

indicates for Firefox at https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html?label=pr_head&max-count=1&pr=19030 ?

@foolip
Copy link
Member Author

foolip commented Nov 28, 2019

https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html?label=pr_head&max-count=1&pr=19030 looks good, the files must be OK because both mp4 and webm pass in Chrome. Looks like Safari doesn't handle resizing mp4 files.

So I'll go ahead and merge this.

@stephenmcgruer FYI, this won't show up as a browser-specific failure because of codec support differing. I don't have any ideas for how to fix this other than changing the query.

@foolip foolip merged commit 3a65d13 into master Nov 28, 2019
@foolip foolip deleted the foolip/resize-during-playback branch November 28, 2019 13:22
@stephenmcgruer
Copy link
Contributor

@stephenmcgruer FYI, this won't show up as a browser-specific failure because of codec support differing. I don't have any ideas for how to fix this other than changing the query.

To be clear, the case you are worried about is that:

i. codec support is not relevant for this test (unknown to me offhand if it's spec'd at all), and
ii. Firefox supports webm but not mp4, and
iii. Safari supports mp4 but not webm, and
iv. Safari does not get the right event sequence for the test.

I think this is probably rare enough to just leave as is, we don't want to start including precondition_failed in browser-specific failure stats. Though I am quite concerned that people are going to use it incorrectly (specifically in cases where a browser doesn't support something that is spec'd and does matter for the test), but time will tell...

@guest271314
Copy link
Contributor

@stephenmcgruer

ii. Firefox supports webm but not mp4, and

Note, Firefox does support mp4 playback.

@stephenmcgruer
Copy link
Contributor

Note, Firefox does support mp4 playback.

Oh, my bad. It seemed to be failing the precondition based on https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html (the precondition is https://github.com/web-platform-tests/wpt/pull/19030/files#diff-785bb71c11bc7b9e3c0fb147340ea8b9R15)

However trying https://output.jsbin.com/vequduj/quiet (which just checks video.canPlayType('video/mp4')) out on Firefox on my Linux desktop I get 'maybe' though (which is acceptable), so seems like something might be weird with the way Firefox is configured in the Taskcluster setup... maybe @gsnedders knows something, or knows who we could ask further?

@guest271314
Copy link
Contributor

"maybe" is not technically equalivalent to failure.

What does

assert_precondition(video.canPlayType(`video/${format}`), `${format} supported`);

expect the output of

video.canPlayType(`video/${format}`)

and

`${format} supported`

to be?

Why is the term "supported" printed in the template literal?

Even Chromium outputs "maybe" for "video/webm" and an empty string "" for "video/x-matroska", although there is a means to both create and playback Matroska videos with H264 and AVC1 codecs at Chromium.

"supported" does not appear to be correct string to test for in any event. Another means can be used to determine if the video can play.

@zcorpan
Copy link
Member

zcorpan commented Dec 2, 2019

canPlayType only gives at best "maybe" when asking for a container format where the codecs are unknown. When codecs parameter is used, and they are also supported, the return value can be "probably".

@guest271314
Copy link
Contributor

Why is the precodition

    function assert_precondition(precondition, description) {
        if (!precondition) {
            throw new PreconditionFailedError(description);
        }
    }

test failing where Firefox 70 and Nightly 72 both output "maybe" for video.canPlayType("video/mp4")?

@zcorpan
Copy link
Member

zcorpan commented Dec 3, 2019

I think @stephenmcgruer is right in

so seems like something might be weird with the way Firefox is configured in the Taskcluster setup...

that is, Firefox as it's run for wpt.fyi doesn't seem to support it. I'm not sure why, or if this is a known thing.
cc @jgraham

@guest271314
Copy link
Contributor

As described at web-platform-tests/wpt.fyi#1648 (comment) there needs to be a means to set and get preferences/flags for browsers run at wpt.fyi. Instead of guessing why certain very basic or experimental features of a given browser are either defined and implemented or not (e.g., OffscreenCanvas tests), after the fact of composing tests for the intended API. Browser settings at wpt.fyi, etc. should be immediately clear before and after the fact.

In the meantime, checking video.canPlayType(<MIME Type>) for an empty string does not necessarily provide accurate results, for example, the case of Matroska support at Chromium, where it is not immediately obvious precisely how to play an .mkv file at Chromium, Chrome.

Am not certain why the assert_precondition function is outputting results that appear to be failure for mp4 at Firefox.

What we do know is that currently Firefox does not support playback of Matroska (.mkv) container (https://bugzilla.mozilla.org/show_bug.cgi?id=1422891; https://bugzilla.mozilla.org/show_bug.cgi?id=1562862) which means we can rely on Firefox throwing a DOMException

    fetch('data:video/x-matroska;base64,<data>')
      .then(r => r.blob())
      .then(blob => {
        const video = document.createElement('video');
        video.controls = true;
        video.src = URL.createObjectURL(blob);
        return video.play()
                  .then(_ => `can play ${blob.type}`)
                  .catch(_ => `cannot play ${blob.type}` /* or empty string to use precondition_assert */);
      })
      .then(console.log, console.error);

DOMException: "The media resource indicated by the src attribute or assigned media provider object was not suitable."

when an .mkv file is played at Firefox or Nightly.

Using that fact we can use the same pattern to test if the video plays by executing play() and returning a value ("can play ") from the new Promise from a chained then(), or catch() the DOMException and return a value ("can not play "), or to continue to use assert_precondition, "+" or "" which should output consistent results at Chrome, Firefox (am not able to verify for Safari) https://plnkr.co/edit/g1rqBtcLwgEC5R1JLQxP?p=preview.

@guest271314
Copy link
Contributor

Note, Firefox will throw an error at play() for .mp4 videos if autoplay for audio and video is not un-blocked.

@guest271314
Copy link
Contributor

Firefox 70 and Nightly 73 pass Run in your browser on w3c-test.org. What does Safari output when the test is run in the browser?

@stephenmcgruer
Copy link
Contributor

There seems to be a lot of cross-talk here. I'm going to try and address what I consider the most pertinent points.

As described at web-platform-tests/wpt.fyi#1648 (comment) there needs to be a means to set and get preferences/flags for browsers run at wpt.fyi. Instead of guessing why certain very basic or experimental features of a given browser are either defined and implemented or not (e.g., OffscreenCanvas tests), after the fact of composing tests for the intended API. Browser settings at wpt.fyi, etc. should be immediately clear before and after the fact.

I do not disagree with your statement. It should be easy for people to understand what flags runs on wpt.fyi were run with. That said, tests should not be written assuming anything about wpt.fyi runs - they should be written to the specs. It is also infeasible to fully describe the environment that leads to a certain result in wpt.fyi; for example in this case I could believe that the behavior we are seeing (Firefox reporting the empty string for canPlayType) is due to some check of the underlying OS or similar rather than an explicit preference given to Firefox.

I have opened #20581 to look into this.

In the meantime, checking video.canPlayType() for an empty string does not necessarily provide accurate results, for example, the case of Matroska support at Chromium, where it is not immediately obvious precisely how to play an .mkv file at Chromium, Chrome.

If Chromium is returning an empty string for mkv, but supports mkv in <video> elements, that sounds like a browser bug to me. The spec is clear about canPlayType:

i. canPlayType is specified to return an empty string if "type is a type that the user agent knows it cannot render"
ii. "a type that the user agent knows it cannot render" is specified as a "type that the user agent knows it cannot render is one that describes a resource that the user agent definitely does not support".

If browsers are widely violating the spec, and we can get agreement that the violation is reasonable, we change the spec. Otherwise we treat deviations from specs as browser bugs.

Using that fact we can use the same pattern to test if the video plays...

Given that canPlayType('video/mp4') is falsey for Firefox on our Taskcluster-based runs, I would suspect that this more complicated approach to detecting support is also going to fail. You could always give it a go though; uploading a pull request with the change will cause a Taskcluster run which should give you an answer.

What does Safari output when the test is run in the browser?

Based on a browserstack run, Safari 13 on Catalina reports "http://w3c-test.org/html/semantics/embedded-content/the-video-element/resize-during-playback.html" for mp4 and PRECONDITION_FAILED for webm. That is the same result as we get from our Azure Pipelines runs of Safari that are shown on wpt.fyi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

HTML: Add test for <video> dispatching resize event and displaying variable video track width and height
7 participants