Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ff132 Relnote: HTMLVideoElement.requestVideoFrameCallback + expr feat #36186

Merged
merged 4 commits into from
Oct 18, 2024
Merged

Ff132 Relnote: HTMLVideoElement.requestVideoFrameCallback + expr feat #36186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
870c915
HTMLVideoElement.requestVideoFrameCallback() - page structure update
hamishwillee Oct 4, 2024
6ff260e
FF132Relnote: HTMLVideoElement.requestVideoFrameCallback + expr feat
hamishwillee Oct 4, 2024
92bd95d
Merge branch 'main' into ff132_relnote_request_vidoe_frame_callback
hamishwillee Oct 11, 2024
38fed57
Merge branch 'main' into ff132_relnote_request_vidoe_frame_callback
hamishwillee Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 0 additions & 40 deletions files/en-us/mozilla/firefox/experimental_features/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1379,46 +1379,6 @@ This enables a fetch request to function as an alternative to {{domxref("Navigat

### Graphics: Canvas, WebGL, and WebGPU

#### Request video frame callbacks

The {{domxref('HTMLVideoElement/requestVideoFrameCallback','requestVideoFrameCallback()')}} method of the {{domxref('HTMLVideoElement')}} interface registers a callback function that runs when a new video frame is sent to the compositor. Developers can use this to perform operations on each video frame, enabling more efficient painting to a canvas, video analysis, synchronization with external audio sources, and so on. The method returns a callback handle that can be passed to {{domxref('HTMLVideoElement.cancelVideoFrameCallback()')}} in order to cancel the outstanding callback request. ([Firefox bug 1800882](https://bugzil.la/1800882)).

<table>
<thead>
<tr>
<th>Release channel</th>
<th>Version added</th>
<th>Enabled by default?</th>
</tr>
</thead>
<tbody>
<tr>
<th>Nightly</th>
<td>130</td>
<td>Yes</td>
</tr>
<tr>
<th>Developer Edition</th>
<td>130</td>
<td>No</td>
</tr>
<tr>
<th>Beta</th>
<td>130</td>
<td>No</td>
</tr>
<tr>
<th>Release</th>
<td>130</td>
<td>No</td>
</tr>
<tr>
<th>Preference name</th>
<td colspan="2"><code>media.rvfc.enabled</code></td>
</tr>
</tbody>
</table>

#### Hit regions

Whether the mouse coordinates are within a particular area on the canvas is a common problem to solve. The hit region API allows you to define an area of your canvas and provides another possibility to expose interactive content on a canvas to accessibility tools.
Expand Down
1 change: 1 addition & 0 deletions files/en-us/mozilla/firefox/releases/132/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ This article provides information about the changes in Firefox 132 that affect d

#### Media, WebRTC, and Web Audio

- The {{domxref('HTMLVideoElement/requestVideoFrameCallback','requestVideoFrameCallback()')}} and {{domxref('HTMLVideoElement/cancelVideoFrameCallback','cancelVideoFrameCallback()')}} methods of the {{domxref('HTMLVideoElement')}} interface are now supported. The `requestVideoFrameCallback()` registers a callback function that runs when a new video frame is sent to the compositor. Developers can use this function to perform operations on each video frame, enabling more efficient painting to a canvas, video analysis, synchronization with external audio sources, and so on. The method returns a callback handle that can be passed to `cancelVideoFrameCallback()` in order to cancel the outstanding callback request. ([Firefox bug 1919367](https://bugzil.la/1919367), [Firefox bug 1800882](https://bugzil.la/1800882)).
- The {{domxref("MediaStreamTrack.getCapabilities()")}} method is now supported. This returns an object detailing the accepted values or value range for each constrainable property of the associated {{domxref("MediaStreamTrack")}} ([Firefox bug 1179084](https://bugzil.la/1179084)).

#### Removals
Expand Down
63 changes: 37 additions & 26 deletions files/en-us/web/api/htmlvideoelement/requestvideoframecallback/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,6 @@ browser-compat: api.HTMLVideoElement.requestVideoFrameCallback

The **`requestVideoFrameCallback()`** method of the {{domxref("HTMLVideoElement")}} interface registers a callback function that runs when a new video frame is sent to the compositor. This enables developers to perform efficient operations on each video frame.

## Description

Typical use cases for `requestVideoFrameCallback()` include video processing and painting to a canvas, video analysis, and synchronization with external audio sources. Per-frame processing used to be done in a less efficient or accurate fashion by running operations on the current video display whenever the {{domxref("HTMLMediaElement.timeupdate_event", "timeupdate")}} event fired. This technique did not provide access to the actual video frames.

`requestVideoFrameCallback()` is used in the same way as {{domxref("Window.requestAnimationFrame()")}}. You use it to run a callback function that performs some operation when the next video frame is sent to the compositor. The callback finishes by calling `requestVideoFrameCallback()` again to run the callback when the next video frame is composited, and so on. However, `requestVideoFrameCallback()` is tailored for video operations in several ways:

- `requestVideoFrameCallback()` provides reliable access to each individual video frame.
- `requestAnimationFrame()` tries to match the display refresh rate, which is typically 60Hz. `requestVideoFrameCallback()`, on the other hand, tries to match the video frame rate. More specifically, the callback will run at the lower of the video frame rate and the browser paint refresh rate. For example, a video with a frame rate of 25fps playing in a browser that paints at 60Hz would fire callbacks at a rate of 25Hz. A video with a frame rate of 120fps running in the same 60Hz browser would fire callbacks at 60Hz.
- `requestVideoFrameCallback()` makes useful video metadata available in the callback function.

One thing to bear in mind is that `requestVideoFrameCallback()` does not offer any strict guarantees that the output from your callback will remain in sync with the video frame rate. It may end up being fired one vertical synchronization (v-sync) later than when the new video frame was presented. (V-sync is a graphics technology that synchronizes the frame rate of a video with the refresh rate of a monitor.)

The API runs on the main thread, while video compositing likely happens on a separate compositing thread. You've got to factor in the time taken for these operations to complete, as well as the time it takes for the video itself and the result of your `requestVideoFrameCallback()` operation to display on the screen.

You can compare the `now` callback parameter and the `expectedDisplayTime` metadata property to determine whether your callback is a v-sync late. If `expectedDisplayTime` is within about five to ten microseconds of `now`, the frame is already rendered. If the `expectedDisplayTime` is approximately sixteen milliseconds in the future (assuming your browser/screen is refreshing at 60Hz), then the callback is a v-sync out.

## Syntax

```js-nolint
Expand All @@ -44,19 +28,30 @@ requestVideoFrameCallback(callback)

- : An object containing the following properties:

- `expectedDisplayTime`: A {{domxref("DOMHighResTimeStamp")}} representing the time when the browser expects the frame to be visible.
- `height`: A number, in media pixels, representing the height of the video frame (the visible decoded pixels, without aspect ratio adjustments).
- `mediaTime`: A number, in seconds, representing the media presentation timestamp of the presented frame. This is equal to the frame's timestamp on the {{domxref("HTMLMediaElement.currentTime")}} timeline.
- `presentationTime`: A {{domxref("DOMHighResTimeStamp")}} representing the time when the browser submitted the frame for composition.
- `presentedFrames`: A number representing the number of frames submitted for composition so far alongside the current callback. This can be used to detect whether frames were missed between callback instances.
- `processingDuration`: A number, in seconds, representing the duration between the submission of the encoded packet with the same presentation timestamp as this frame to the decoder (i.e., the `mediaTime`) and the decoded frame being ready for presentation.
- `width`: A number, in media pixels, representing the width of the video frame (the visible decoded pixels, without aspect ratio adjustments).
- `expectedDisplayTime`
- : A {{domxref("DOMHighResTimeStamp")}} representing the time when the browser expects the frame to be visible.
- `height`
- : A number, in media pixels, representing the height of the video frame (the visible decoded pixels, without aspect ratio adjustments).
- `mediaTime`
- : A number, in seconds, representing the media presentation timestamp of the presented frame. This is equal to the frame's timestamp on the {{domxref("HTMLMediaElement.currentTime")}} timeline.
- `presentationTime`
- : A {{domxref("DOMHighResTimeStamp")}} representing the time when the browser submitted the frame for composition.
- `presentedFrames`
- : A number representing the number of frames submitted for composition so far alongside the current callback.
This can be used to detect whether frames were missed between callback instances.
- `processingDuration`
- : A number, in seconds, representing the duration between the submission of the encoded packet with the same presentation timestamp as this frame to the decoder (i.e., the `mediaTime`) and the decoded frame being ready for presentation.
- `width`
- : A number, in media pixels, representing the width of the video frame (the visible decoded pixels, without aspect ratio adjustments).

Additional metadata properties may be available within `requestVideoFrameCallback()` callbacks used in {{domxref("WebRTC_API", "WebRTC", "", "nocode")}} applications:

- `captureTime`: A {{domxref("DOMHighResTimeStamp")}} representing the time when the frame was captured. This applies to video frames coming from a local or remote source. For a remote source, the capture time is estimated using clock synchronization and RTCP sender reports to convert RTP timestamps to capture time.
- `receiveTime`: A {{domxref("DOMHighResTimeStamp")}} representing the time when the encoded frame was received by the platform. This applies to video frames coming from a remote source. Specifically, this corresponds to the time when the last packet belonging to this frame was received over the network.
- `rtpTimestamp`: A number representing the RTP timestamp associated with this video frame.
- `captureTime`
- : A {{domxref("DOMHighResTimeStamp")}} representing the time when the frame was captured. This applies to video frames coming from a local or remote source. For a remote source, the capture time is estimated using clock synchronization and RTCP sender reports to convert RTP timestamps to capture time.
- `receiveTime`
- : A {{domxref("DOMHighResTimeStamp")}} representing the time when the encoded frame was received by the platform. This applies to video frames coming from a remote source. Specifically, this corresponds to the time when the last packet belonging to this frame was received over the network.
- `rtpTimestamp`
- : A number representing the RTP timestamp associated with this video frame.

> **Note:** `width` and `height` may differ from {{domxref("HTMLVideoElement.videoWidth")}} and {{domxref("HTMLVideoElement.videoHeight")}} in certain cases (for example, an anamorphic video may have rectangular pixels).

Expand All @@ -66,6 +61,22 @@ A number representing a unique callback ID.

This can be passed to {{DOMxRef("HTMLVideoElement.cancelVideoFrameCallback()")}} to cancel the callback registration.

## Description

Typical use cases for `requestVideoFrameCallback()` include video processing and painting to a canvas, video analysis, and synchronization with external audio sources. Per-frame processing used to be done in a less efficient or accurate fashion by running operations on the current video display whenever the {{domxref("HTMLMediaElement.timeupdate_event", "timeupdate")}} event fired. This technique did not provide access to the actual video frames.

`requestVideoFrameCallback()` is used in the same way as {{domxref("Window.requestAnimationFrame()")}}. You use it to run a callback function that performs some operation when the next video frame is sent to the compositor. The callback finishes by calling `requestVideoFrameCallback()` again to run the callback when the next video frame is composited, and so on. However, `requestVideoFrameCallback()` is tailored for video operations in several ways:

- `requestVideoFrameCallback()` provides reliable access to each individual video frame.
- `requestAnimationFrame()` tries to match the display refresh rate, which is typically 60Hz. `requestVideoFrameCallback()`, on the other hand, tries to match the video frame rate. More specifically, the callback will run at the lower of the video frame rate and the browser paint refresh rate. For example, a video with a frame rate of 25fps playing in a browser that paints at 60Hz would fire callbacks at a rate of 25Hz. A video with a frame rate of 120fps running in the same 60Hz browser would fire callbacks at 60Hz.
- `requestVideoFrameCallback()` makes useful video metadata available in the callback function.

One thing to bear in mind is that `requestVideoFrameCallback()` does not offer any strict guarantees that the output from your callback will remain in sync with the video frame rate. It may end up being fired one vertical synchronization (v-sync) later than when the new video frame was presented. (V-sync is a graphics technology that synchronizes the frame rate of a video with the refresh rate of a monitor.)

The API runs on the main thread, while video compositing likely happens on a separate compositing thread. You've got to factor in the time taken for these operations to complete, as well as the time it takes for the video itself and the result of your `requestVideoFrameCallback()` operation to display on the screen.

You can compare the `now` callback parameter and the `expectedDisplayTime` metadata property to determine whether your callback is a v-sync late. If `expectedDisplayTime` is within about five to ten microseconds of `now`, the frame is already rendered. If the `expectedDisplayTime` is approximately sixteen milliseconds in the future (assuming your browser/screen is refreshing at 60Hz), then the callback is a v-sync out.

## Examples

### Drawing video frames on a canvas
Expand Down