Related w3c/csswg-drafts#4266

@foolip How to determine which specification owns resizing of HTMLVideoElement?

@guest271314 that's defined in https://html.spec.whatwg.org/multipage/media.html, searcg for "named resize" to find the two places the event is dispatched.

One way to find the relevant spec is by following spec links at the bottom of MDN pages, in this case https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement.

Regarding the topic at hand, I didn't know it was even possible to construct a WebM video with variable dimensions, since the width and height is part of the metadata in the beginning of the file. It is of course possible to put video of a different size in the file and for the size to change at any keyframe, but is this a valid WebM file?

Ogg is a format where the up-front metadata is optional and which I think would also trigger this problem.

@foolip Yes, the WebM file is valid, in brief see https://gitlab.com/mbunkus/mkvtoolnix/issues/2582#note_189520756; https://mkvtoolnix.download/doc/NEWS.md

Version 36.0.0 "Is That Jazz?" 2019-08-10

New features and enhancements

• mkvmerge: mkvmerge now allows appending AV1, VP8, VP9, H.264/AVC and
  H.265/HEVC tracks whose pixel dimensions differ. Implements Update battery-plugging-in-manual.html #2582.

It is argued that H264 and AVC1 codecs are invalid for WebM files, though that has not stopped Chromium authors from implemented the codec for MediaRecorder https://bugs.chromium.org/p/chromium/issues/detail?id=980822.

In fact, the only way that have been able to get Chromium, using a WebM file encoded at Chromium, to display variable pixel dimensions at Chromium is to use H264 or AVC1 codecs (openh264).

Alright, I would have thought that producing a video stream like that using VP8 or VP9 would be straightforward, but maybe the decoder rejects it?

@foolip The issue is that Chromium does not display the variable pixel dimension video at HTML <video> that its own implementation of MediaRecorder outputs.

This https://plnkr.co/edit/Axkb8s?p=info is the only code that have been able to compose which outputs variable pixel dimensions at HTML <video> element at Chromium browser https://plnkr.co/edit/Axkb8s?p=info. Since the codecs are "non-standard" for a WebM file and Mozilla does not currently decode Matroska files the video does not play at Firefox or Nightly.

This demonstrates the issue with HTML <video> element at Chromium even when the input video is VP8 or VP9 and srcObject of a <video> is set to a MediaStream where the media source is a video created by libwebm https://plnkr.co/edit/sMA653?p=info; et. al.

If you follow the links in the issues you will see that it is not immediately clear which specification owns and is responsible for HTML <video> element resizing.

The decoder at Chromium apparently only reads the initial frame dimensions when the codec is VP8 or VP9, though not for H264 or AVC1.

Again, when trying to determine which specification owns and controls resizing of HTML <video> element (to file issues at the appropriate specification, where am not banned from, in order to mandate that the <video> is resized), the answer is not immediately clear.

@foolip Re

I didn't know it was even possible to construct a WebM video with variable dimensions

see https://github.com/guest271314/native-messaging-mkvmerge.

@foolip For more details and to further demonstrate the issue see https://bugs.chromium.org/p/chromium/issues/detail?id=983777#c20, https://bugs.chromium.org/p/chromium/issues/detail?id=983777#c21, https://bugs.chromium.org/p/chromium/issues/detail?id=983777#c22.

@foolip Interestingly Chromium does display variable pixel dimensions and dispatch resize event for a WebM video recorded with MediaRecorder at Mozilla with codecs set to VP8. Diving in to Chromium source code there are sections which evaluate pixel dimensions and frame rate https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/mediarecorder/media_recorder_handler.cc?dr=CSs&g=0&l=38 (see also https://bugs.chromium.org/p/webm/issues/detail?id=1642.) which could explain (though not sure exactly how) the output at https://plnkr.co/edit/4JxS4O?p=info using webm-writer.js
https://github.com/thenickdude/webm-writer-js.

The consequence is that videoWidth and videoHeight of HTML <video> element (and MedisStreamTrack of kind "video" width and height from getSettings(), to the extent applicable where getSettings() is defined for captureStream() in specifications that use and refere toMediaStream though are not specifically Media Capture and Streams https://bugzilla.mozilla.org/show_bug.cgi?id=1537986) values are not accurate relevant to the actual underlying encoded frame during playback at Chromium, e.g., https://plnkr.co/edit/uCnx1R?p=info.

It has been somewhat challenging to determine which specification and implemented API is responsible for the issue of resizing HTML <video> element at Chromium; MediaRecorder; HTML <video> element; or CSS?

Presently am not certain by reading the current iteration of the relevant specifications; even the new ResizeObserver API does not explicitly state that it owns resizing HTML elements; i.e., from w3c/csswg-drafts#4266

• https://www.w3.org/TR/2014/WD-DOM-Level-3-Events-20140925/#event-type-resize (1 occurence of the text "resize")
• https://drafts.csswg.org/cssom/#event-type-resize (0 occurence of the text "resize")

It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here. HTML together with container format specs like WebM are the specs that define what should happen.

If both WebM and MPEG-4 support resizing video streams, then adding a test for this would be straightforward.

@foolip

It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here.

Do we disregard or edit the language at MDN https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event?

However, resize events are only fired on the window object (i.e. returned by document.defaultView). Only handlers registered on the window object will receive resize events.

Am currently banned (evidently indefinitely) from WHATWG. Am still willing to help or write the tests if a member of WPT files the PR.

Yes, https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event should be disregarded for two reasons. First, its concerns the resize event fired at the window object, which isn't defined in the same place as the resize event fired at media elements. Second, nothing MDN says has any bearing on what browsers should implement or what wpt should test, since it's documentation intended for web developers.

If you can produce media files for this, then getting a test into wpt should be straightforward.

@foolip

If you can produce media files for this, then getting a test into wpt should be straightforward.

guest271314/MediaFragmentRecorder@ca44c12

I guess MediaRecorder only supports WebM/Matroska, but is it possible to have an MPEG-4 container with video that changes its size mid-stream?

The ideal test video files would be one that is red and 400x300 for 1 second, and then green and 300x150 for 1 second. The exact size isn't very important, but something smallish is good. The size of the videos in bytes should be as small as possible, it should be possible to get them below 100kb I think?

Large test files makes the tests slower and makes it more likely that failures are due to something on the network side rather than the media decoding pipeline.

I guess MediaRecorder only supports WebM/Matroska, but is it possible to have an MPEG-4 container with video that changes its size mid-stream?

Cannot fathom why that would not be possible. Would that be dissimilar from the Matroska container width H264 (openh264) codec at the previous link?

The ideal test video files would be one that is red and 400x300 for 1 second, and then green and 300x150 for 1 second. The exact size isn't very important, but something smallish is good. The size of the videos in bytes should be as small as possible, it should be possible to get them below 100kb I think?

Created the various files that have previously tested to demonstrate the issue with interoperability. The variation of the media files and display of those files at HTML <video> element output by the browsers. There are workarounds to acheieve the expected result at both browsers, though as yet
not using the same code. Use them or not as the individual test description requires. The plnkr https://plnkr.co/edit/uCnx1R?p=preview at the linked Chromium issue https://bugs.chromium.org/p/chromium/issues/detail?id=972470 provides a means to create the file chromium_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip

@foolip

      const imageData = [{
        width: 400,
        height: 300,
        fillStyle: "red"
      }, {
        width: 300,
        height: 150,
        fillStyle: "green"
      }];

Compare the output of
firefox_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip at Chromium to the output of the WebM file produced by the same code at Chromium. The WebM file produced at Chromium does not resize at HTML <video> element at Chromium, does resize at Firefox. The WebM file produced Firefox resizes HTML <video> element at Chromium and Firefox.

There are a lot of video files flying around now :) Is there a pair of small video files, one WebM and the other something that Safari supports, that could be used to test the resize event behavior? Please point directly to which ones you're proposing.

The WebM with VP8 codec file is in the .zip archive at previous post.
chromium_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.mp4.zip. Do not have access to Safari to test playback. The video was recorded at Chromium using h264 codec the converted to MP4 using this command ffmpeg -i video.webm -c:v copy video.mp4 (Converting WebM to MP4 Using FFmpeg).

This was recorded at Firefox https://github.com/web-platform-tests/wpt/files/3588824/firefox_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip and should dispatch resize event at Chromium. The file recorded at Chromium (at this post #17821 (comment)) currently will not dispatch resize event at Chromium (TL;DR https://bugs.chromium.org/p/chromium/issues/detail?id=1001948).

Cool, thanks! Putting that in a video element and playing in Chrome, I actually do see resize events. I don't think this is tested in wpt however, so turning it into a test is a good idea. I don't have time to finish it now, but #19030 is the structure for a test.

Alright, I finished #19030, but it seems to work in Chrome, so not covering what you'd hoped for I think.

@foolip

but it seems to work in Chrome, so not covering what you'd hoped for I think.

Do not entertain "hope".

The expected result is for the video produced by Chromium to resize at Chromium.

You must have used the WebM file recorded at Firefox for the test, correct?

The .webm video in the test was created at Firefox

$ mkvmerge -J 400x300-red-resize-300x150-green.webm
{
  "attachments": [],
  "chapters": [],
  "container": {
    "properties": {
      "container_type": 17,
      "is_providing_timestamps": true,
      "muxing_application": "QTmuxingAppLibWebM-0.0.1",
      "writing_application": "QTwritingAppLibWebM-0.0.1"
    },
    "recognized": true,
    "supported": true,
    "type": "Matroska"
  },
  "errors": [],
  "file_name": "400x300-red-resize-300x150-green.webm",
  "global_tags": [],
  "identification_format_version": 12,
  "track_tags": [],
  "tracks": [
    {
      "codec": "VP8",
      "id": 0,
      "properties": {
        "codec_id": "V_VP8",
        "codec_name": "VP8",
        "codec_private_length": 0,
        "default_track": true,
        "display_dimensions": "400x300",
        "display_unit": 0,
        "enabled_track": true,
        "forced_track": false,
        "language": "eng",
        "minimum_timestamp": 0,
        "number": 1,
        "pixel_dimensions": "400x300",
        "uid": 403731017
      },
      "type": "video"
    }
  ],
  "warnings": []
}

Have you tried running the test with the WebM file generated by Chromium implementation of MediaRecorder included at #17821 (comment)?

I'll have to double-check, I think I used the file in #17821 (comment)

The file was generated by Firefox https://bugzilla.mozilla.org/show_bug.cgi?id=1177767, see

      "muxing_application": "QTmuxingAppLibWebM-0.0.1",
      "writing_application": "QTwritingAppLibWebM-0.0.1"

For a comprehensive test would include the file produced by Chrome as well.

@foolip

Alright, I finished #19030, but it seems to work in Chrome, so not covering what you'd hoped for I think.

This demonstrates the issue at Chromium https://plnkr.co/edit/n1rGNe?p=info.

@foolip FWIW Proposed a specification change to HTML <video> at https://discourse.wicg.io/t/proposal-htmlvideoelement-decode-and-display-frame-resolution/3933.

@foolip

It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here.

From your reading of HTML Standard does the specification

Whenever the intrinsic width or intrinsic height of the video changes (including, for example, because the selected video track was changed), if the element's readyState attribute is not HAVE_NOTHING, the user agent must queue a task to fire an event named resize at the media element.

mandate that resize event must be fired when the intrinsic width or intrinsic height - in this case the fact that the developer encodes variable resolution frames directly affecting intrinsic width and intrinsic height?

Or, does the HTML specification (WHATWG or W3C) not mandate that the resize event MUST be dispatched when the underlying encoded frame resolution (intrinsic width and intrinsic height) changes, for example, from 400x300 to 300x150?

The resize event should be fired when the intrinsic size changes, but it comes down to the definition of intrinsic width and intrinsic height, which is ultimately a question for the container and video format. I don't know much about this, but there are probably video codecs that internally downscale to improve quality/bit, where a correct player should upscale the video frames. This is somewhat similar to the video having a different aspect ratio than the raw video frames.

So, what should happen with the video streams you've produced depends on other specs that HTML, but my best guess is that resizing and firing a resize event is the right thing.

@foolip The reason asked the question are the contents of this comment https://bugs.chromium.org/p/chromium/issues/detail?id=992235#c29

The change means that it is not possible to display variable resolution video at Chromium using the Blob that Chromium produced at MediaRecorder

Variable resolution videos can still be played, but the video element will not dynamically resize.

The HTML

This is working as expected. The video plays through ffmpeg unless you're part of the 1% libvpx experiment. It probably also plays through ffmpeg in firefox. If we launch the libvpx experiment it will cease to do resolution changes in Chrome.

and specifically 3)

It is reasonable to expect that variable resolution input frames to MediaStreamTrack and recorded with MediaRecorder should display the underlying encoded frames at HTML

I agree with the high level statement, but I the details of paint a different picture.

1. Given the crashes are CHECKs, its unclear that MediaRecorder really supports this use case (I ack that it works for some codecs)
2. Even when an implicit resolution change file could be produced by MediaRecorder, we will still play it back, we just may not resize the element for the later resolutions.
3. No spec mandates that we change the video size for implicit resolution changes, and we are already mostly not doing it for other codecs.

which implies that HTML does not consider or currently unambiguously define "intrinsic width" and "intrinsic height" as encompassing variable resolution frames encoded in the container which the developer is reasonably being expected to be decoded and displayed at HTML <video> element, else the end-user would not have encoded the variable resolution frames. From perspective here intentionally encoded variable resolution frames MUST be included within the defintion of "intrinsic width" and "intrinsic height" - there is notthing else an "intrinsic width" could refer to relevant to updating videoWidth and firing resize event. If the terms means something else which excludes the end-user intentionally encoding variable resolution frames to the container that needs to be clarified at HTML specification.

The containers Matroska and WebM and the codecs VP8 and VP9 do support encoding variable resolution frames. There is no dispute of facts there. The issue is the arbitrary decision by implementer of HTML <video> to prevent decoding of variable resolution frames and the display those frames at <video> element - perhaps for the purpose of targeting certain devices.

The question is simple: where the specification currently states

the user agent must queue a task to fire an event named resize at the media element.

do the terms "intrinsic width" and "intrinsic height" in the HTML specification apply to the underlying variable resolution encoded video frames?