Summary

Problem Overview

Recall that applications may currently obtain a capture of the tab in which they run using getDisplayMedia, either with or without preferCurrentTab. Moreover, soon another API will allow similar functionality - getViewportMedia. In either case, the application may then also wish to crop the resulting video track so as to remove some content from it (typically before sharing it remotely). We introduce a performant and robust API for cropping a self-capture video track.

Core Challenges

Layout can change asynchronously when the user scrolls, zooms or resizes the window. The application cannot robustly react to such changes without risking mis-cropping the video track on occasion. The browser therefore needs to step in and help.

Sample Use Case

Consider a combo-application consisting of two major parts - a video-conferencing application and a productivity-suite application co-existing in a single tab. Assume the video-conferencing uses existing/upcoming APIs such as getDisplayMedia and/or getViewportMedia and captures the entire tab. Now it needs to crop away everything other than a particular section of the productivity-suite. It needs to crop away its own video-conferencing content, any speaker notes and other private and/or irrelevant content in the productivity-suite, before transmitting the resulting cropped video remotely.

Moreover, consider that it is likely that the two collaborating applications are cross-origin from each other. They can post messages, but all communication is asynchronous, and it's easier and more performant if information is transmitted sparingly between them. That precludes solutions involving posting of entire frames, as well as solutions which are too slow to react to changes in layout (e.g. scrolling, zooming and window-size changes).

Goals and Non-Goals

Goals

• The new API we introduce allows an application which is already in possession of a self-capture video track, to crop that track to the contours of its desired element.
• The API allows this to be done performantly, consistently and robustly.

Non-Goals

• This API does not introduce new ways to obtain a self-capture video track.
• This API does not introduce mechanisms by which a captured document may control what the capturing document can see.

Solution

Solution Overview

A two-pronged solution is presented:

• CropTarget production: A mechanism for tagging an HTMLElement as a potential target for the cropping mechanism.
• Cropping mechanism: A mechanism for instructing the user agent to start cropping a video track to the contours of a previously tagged HTMLElement, or to stop such cropping and revert a track to its uncropped state.

CropTarget production

We introduce navigator.mediaDevices.produceCropTarget().

partial interface MediaDevices {
  Promise<CropTarget>
  produceCropTarget(HTMLElement target);
};

Given an HTMLElement, produceCropTarget() produces an opaque class that uniquely identifies that element to our second mechanism - the cropping mechanism. (The Promise returned by produceCropTarget() is only resolved when the CropTarget is ready for use, allowing the browser time to set up prerequisites and propagate state cross-process.)

Cropping mechanism

We introduce a cropTo() method, which we expose on all video tracks derived of tab-capture.

[Exposed = Window]
interface BrowserCaptureMediaStreamTrack : FocusableMediaStreamTrack {
  Promise<undefined> cropTo(CropTarget cropTarget);
};

Given a CropTarget, cropTo() starts cropping the video track to the contours of the referenced HTMLElement. Given undefined, cropTo() reverts a video track to its uncropped state. "On-the-fly" changing of crop-targets is possible.

Code Samples

/////////////////////////////////
// Code in the capture-target: //
/////////////////////////////////

const mainContentArea = navigator.getElementById('mainContentArea');
const cropTarget = await navigator.mediaDevices.produceCropTarget(mainContentArea);
sendCropTarget(cropTarget);

function sendCropTarget(cropTarget) {
  // Can send the crop-target to another document in this browsing context
  // using postMessage() or using any other means.
  // Possibly there is no other document, and this is just consumed locally.
}

/////////////////////////////////////
// Code in the capturing-document: //
/////////////////////////////////////

async function startCroppedCapture(cropTarget) {
  const stream = await navigator.mediaDevices.getDisplayMedia();
  const [track] = stream.getVideoTracks();
  if (!!track.cropTo) {
    handleError(stream);
    return;
  }
  await track.cropTo(cropTarget);
  transmitVideoRemotely(track);
}

Changelog

See dedicated file.