plane-detection.bs

<pre class="metadata">
Shortname: webxr-plane-detection
Title: WebXR Plane Detection Module
Group: immersivewebcg
Status: CG-DRAFT
Level: 1
ED: https://github.com/immersive-web/real-world-geometry/
Repository: immersive-web/real-world-geometry
Mailing List Archives: https://lists.w3.org/Archives/Public/public-immersive-web/

!Participate: <a href="https://github.com/immersive-web/real-world-geometry/issues/new">File an issue</a> (<a href="https://github.com/immersive-web/real-world-geometry/issues">open issues</a>)
!Participate: <a href="https://lists.w3.org/Archives/Public/public-immersive-web/">Mailing list archive</a>
!Participate: <a href="irc://irc.w3.org:6665/">W3C's #immersive-web IRC</a>

Editor: Piotr Bialecki 114482, Google http://google.com/, bialpio@google.com

Abstract:
</pre>

<pre class="link-defaults">
</pre>

<pre class="anchors">
spec: WebXR Device API - Level 1; urlPrefix: https://immersive-web.github.io/webxr/#
    for: XRFrame;
        type: dfn; text: active; url: xrframe-active
        type: dfn; text: session; url: dom-xrframe-session
        type: dfn; text: time; url: xrframe-time
    for: XRSession;
        type: dfn; text: list of frame updates; url: xrsession-list-of-frame-updates
        type: dfn; text: mode; url: xrsession-mode
        type: dfn; text: XR device; url: xrsession-xr-device
    for: XRSpace;
        type: dfn; text: effective origin; url: xrspace-effective-origin
        type: dfn; text: native origin; url: xrspace-native-origin
        type: dfn; text: origin offset; url: xrspace-origin-offset
        type: dfn; text: session; url: xrspace-session
    type: dfn; text: capable of supporting; url: capable-of-supporting
    type: dfn; text: feature descriptor; url: feature-descriptor
    type: dfn; text: identity transform; url: identity-transform
    type: dfn; text: inline XR device; url: inline-xr-device
    type: dfn; text: quantization; url: quantization
    type: dfn; text: rounding; url: rounding
    type: dfn; text: XR device; url: xr-device
spec: WebXR Anchors Module; urlPrefix: https://immersive-web.github.io/anchors/#
    type: dfn; text: create new anchor object; url: create-new-anchor-object
    type: dfn; text: update anchors; url: update-anchors
    type: interface; text: XRAnchor; url: xr-anchor
</pre>

<pre class=biblio>
{
  "webxr-anchors-module": {
    "authors": [
      "Piotr Bialecki"
    ],
    "href": "https://immersive-web.github.io/anchors/",
    "title": "WebXR Anchors Module",
    "status": "DR"
  }
}
</pre>

<style>
  .non-normative::before {
    content: "This section is non-normative.";
    font-style: italic;
  }
  .tg {
    border-collapse: collapse;
    border-spacing: 0;
  }
  .tg th {
    border-style: solid;
    border-width: 1px;
    background: #704822;
    color: #fff;
    font-family: sans-serif;
    font-weight: bold;
    border-color: grey;
  }
  .tg td {
    padding: 4px 5px;
    background-color: rgb(30, 17, 0);
    font-family: monospace;
    border-style: solid;
    border-width: 1px;
    border-color: grey;
    overflow: hidden;
    word-break: normal;
  }
  .unstable::before {
    content: "This section is not stable";
    display: block;
    font-weight: bold;
    text-align: right;
    color: red;
  }
  .unstable {
    border: thin solid pink;
    border-radius: .5em;
    padding: .5em;
    margin: .5em calc(-0.5em - 1px);
    background-image: url("data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' width='300' height='290'><text transform='rotate(-45)' text-anchor='middle' font-family='sans-serif' font-weight='bold' font-size='70' y='210' opacity='.1' fill='white'>Unstable</text></svg>");
    background-repeat: repeat;
    background-color: #282828;
  }
  @media (prefers-color-scheme: light) {
    .unstable {
      background-image: url("data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' width='300' height='290'><text transform='rotate(-45)' text-anchor='middle' font-family='sans-serif' font-weight='bold' font-size='70' y='210' opacity='.1'>Unstable</text></svg>");
      background-color: #FFF4F4;
    }

    .tg th {
      background: #90b8de;
    }

    .tg td {
      background-color: rgb(221, 238, 255);
    }
  }
  .unstable h3:first-of-type {
    margin-top: 0.5rem;
  }
  .unstable.example:not(.no-marker)::before {
    content: "Example " counter(example) " (Unstable)";
    float: none;
  }
</style>

Introduction {#intro}
============

<section class="non-normative">

</section>

Initialization {#anchor-feature-initialization}
==================

Feature descriptor {#anchor-feature-descriptor}
------------------

In order for the applications to signal their interest in using plane detection during a session, the session must be requested with appropriate [=feature descriptor=]. The string <dfn>plane-detection</dfn> is introduced by this module as a new valid feature descriptor for plane detection feature.

A device is [=capable of supporting=] the plane-detection feature if the device's tracking system exposes a [=native plane detection=] capability. The [=inline XR device=] MUST NOT be treated as [=capable of supporting=] the plane-detection feature.

When a session is created with plane-detection feature enabled, the [=update planes=] algorithm MUST be added to the [=list of frame updates=] of that session.

<div class="example">
The following code demonstrates how a session that requires plane detection could be requested:

<pre highlight="js">
const session = await navigator.xr.requestSession("immersive-ar", {
  requiredFeatures: ["plane-detection"]
});
</pre>

</div>

Planes {#planes-section}
======

XRPlaneOrientation {#plane-orientation}
------------------

<script type="idl">
enum XRPlaneOrientation {
    "horizontal",
    "vertical"
};
</script>

- A plane orientation of <dfn enum-value for="XRPlaneOrientation">"horizontal"</dfn> indicates that the plane is primarily oriented horizontally (according to the conventions of the underlying platform).
- A plane orientation of <dfn enum-value for="XRPlaneOrientation">"vertical"</dfn> indicates that the plane is primarily oriented vertically (according to the conventions of the underlying platform).

XRPlane {#plane}
-------

<script type="idl">
interface XRPlane {
    [SameObject] readonly attribute XRSpace planeSpace;

    readonly attribute FrozenArray<DOMPointReadOnly> polygon;
    readonly attribute XRPlaneOrientation? orientation;
    readonly attribute DOMHighResTimeStamp lastChangedTime;
    readonly attribute DOMString? semanticLabel;
};
</script>

An {{XRPlane}} represents a single, flat surface detected by the underlying XR system.

The {{XRPlane/planeSpace}} is an {{XRSpace}} that establishes the coordinate system of the plane. The [=XRSpace/native origin=] of the {{XRPlane/planeSpace}} tracks plane's center. The underlying XR system defines the exact meaning of the plane center. The Y axis of the coordinate system defined by {{XRPlane/planeSpace}} MUST represent the plane's normal vector.

Each {{XRPlane}} has an associated <dfn for=XRPlane>native entity</dfn>.

Each {{XRPlane}} has an associated <dfn for=XRPlane>frame</dfn>.

The {{XRPlane/polygon}} is an array of vertices that describe the shape of the plane. They are returned in the form of a loop of points at the edges of the polygon, expressed in the coordinate system defined by {{XRPlane/planeSpace}}. The Y coordinate of each vertex MUST be <code>0.0</code>.

<div class="unstable">

The {{XRPlane/semanticLabel}} attribute is a string that describes the [=semantic label=] of the polygon. This array is empty if there is no semantic information. The {{XRSystem}} SHOULD populate this  with the [=semantic label=] it has knowledge of.

A <dfn>semantic label</dfn> is an ASCII lowercase DOMString that describes the name in the real world of the {{XRPlane}} as known by the {{XRSystem}}.
The list of semantic labels is defined in the <a href="https://github.com/immersive-web/semantic-labels">semantic label registry</a>.

</div>

The {{XRPlane/orientation}} describes orientation of the plane, as classified by the underlying XR system. In case the orientation cannot be classified into {{XRPlaneOrientation/"horizontal"}} or {{XRPlaneOrientation/"vertical"}} by the underlying XR system, this attribute will be set to <code>null</code>.

The {{XRPlane/lastChangedTime}} is the last time some of the plane attributes have been changed.

Note: The pose of a plane is not considered a plane attribute and therefore updates to plane pose will not cause the {{XRPlane/lastChangedTime}} to change. This is because plane pose is a property that is derived from two different entities - {{XRPlane/planeSpace}} and the {{XRSpace}} relative to which the pose is to be computed via {{XRFrame/getPose()}} function.

Obtaining detected planes {#obtaining-planes}
=========================

XRPlaneSet {#plane-set}
----------

<script type="idl">
interface XRPlaneSet {
  readonly setlike<XRPlane>;
};
</script>

An {{XRPlaneSet}} is a collection of {{XRPlane}}s. It is the primary mechanism of obtaining the collection of planes detected in an {{XRFrame}}.

<script type="idl">
partial interface XRFrame {
  readonly attribute XRPlaneSet detectedPlanes;
};
</script>

{{XRFrame}} is extended to contain {{XRFrame/detectedPlanes}} attribute which contains all planes that are still tracked in the frame. The set is initially empty and will be populated by the [=update planes=] algorithm. If this attribute is accessed when the frame is not [=XRFrame/active=], the user agent MUST throw {{InvalidStateError}}.

<div class="unstable">

<script type="idl">
partial interface XRSession {
  Promise<undefined> initiateRoomCapture();
};
</script>

{{XRSession}} is extended to contain the {{XRSession/initiateRoomCapture}} method which, if supported, will ask the {{XR Compositor}} to capture the current room layout. It is up to the {{XRCompositor}} if this will replace or augment the [=XRSession/set of tracked planes=]. The user agent MAY also ignore this call, for instance if it doesn't support a manual room capture more or if it determines that the room is already set up.
The {{XRSession/initiateRoomCapture}} method MUST only be able to be called once per {{XRSession}}.

</div>

{{XRSession}} is also extended to contain associated <dfn for=XRSession>set of tracked planes</dfn>, which is initially empty. The elements of the set will be of {{XRPlane}} type.

<div class="algorithm" data-algorithm="update-planes">
In order to <dfn>update planes</dfn> for |frame|, the user agent MUST run the following steps:
    1. Let |session| be a |frame|'s [=XRFrame/session=].
    1. Let |device| be a |session|'s [=XRSession/XR device=].
    1. Let |trackedPlanes| be a result of calling into |device|'s [=native plane detection=] capability to obtain tracked planes at |frame|'s [=XRFrame/time=].
    1. For each |native plane| in |trackedPlanes|, run:
        1. If desired, treat the |native plane| as if it were not present in |trackedPlanes| and continue to the next entry. See [[#privacy-security]] for criteria that could be used to determine whether an entry should be ignored in this way.
        1. If |session|'s [=XRSession/set of tracked planes=] contains an object |plane| that [=corresponds to=] |native plane|, invoke [=update plane object=] algorithm with |plane|, |native plane|, and |frame|, and continue to the next entry.
        1. Let |plane| be the result of invoking the [=create plane object=] algorithm with |native plane| and |frame|.
        1. Add |plane| to |session|'s [=XRSession/set of tracked planes=].
    1. Remove each object in |session|'s [=XRSession/set of tracked planes=] that was neither created nor updated during the invocation of this algorithm.
    1. Set |frame|'s {{XRFrame/detectedPlanes}} to [=XRSession/set of tracked planes=].
</div>

<div class="algorithm" data-algorithm="create-plane-object">
In order to <dfn>create plane object</dfn> from a [=native plane object=] |native plane| and {{XRFrame}} |frame|, the user agent MUST run the following steps:
    1. Let |result| be a new instance of {{XRPlane}}.
    1. Set |result|'s [=XRPlane/native entity=] to |native plane|.
    1. Set |result|'s {{XRPlane/planeSpace}} to a new {{XRSpace}} object created with [=XRSpace/session=] set to |frame|'s {{XRFrame/session}} and [=XRSpace/native origin=] set to track |native plane|'s native origin.
    1. Invoke [=update plane object=] algorithm with |result|, |native plane|, and |frame|.
    1. Return |result|.

A plane object, |result|, created in such way is said to <dfn>correspond to</dfn> the passed in native plane object |native plane|.
</div>

<div class="algorithm" data-algorithm="update-plane-object">
In order to <dfn>update plane object</dfn> |plane| from a [=native plane object=] |native plane| and {{XRFrame}} |frame|, the user agent MUST run the following steps:
    1. Set |plane|'s [=XRPlane/frame=] to |frame|.
    1. If |native plane| is classified by the underlying system as vertical, set |plane|'s {{XRPlane/orientation}} to {{XRPlaneOrientation/"vertical"}}. Otherwise, if |native plane| is classified by the underlying system as horizontal, set |plane|'s {{XRPlane/orientation}} to {{XRPlaneOrientation/"horizontal"}}. Otherwise, set |plane|'s {{XRPlane/orientation}} to <code>null</code>.
    1. Set |plane|'s {{XRPlane/polygon}} to the new array of vertices representing |native plane|'s polygon, performing all necessary conversions to account for differences in native plane polygon representation.
    1. <div class="unstable">Set |plane|'s {{XRPlane/semanticLabel}} to a new string with the [=semantic label|semantic labels=].</div>
    1. If desired, reduce the level of detail of the |plane|'s {{XRPlane/polygon}} as described in [[#privacy-security]].
    1. Set |plane|'s {{XRPlane/lastChangedTime}} to [=XRFrame/time=].
</div>

<div class="example">

The following example demonstrates how an application could obtain information about detected planes and act on it. The code that can be used to render a graphical representation of the planes is not shown.

<pre highlight="js">
// `planes` will track all detected planes that the application is aware of,
// and at what timestamp they were updated. Initially, this is an empty map.
const planes = Map();

function onXRFrame(timestamp, frame) {
  const detectedPlanes = frame.detectedPlanes;

  // First, let's check if any of the planes we knew about is no longer tracked:
  for (const [plane, timestamp] of planes) {
    if(!detectedPlanes.has(plane)) {
      // Handle removed plane - `plane` was present in previous frame,
      // but is no longer tracked.

      // We know the plane no longer exists, remove it from the map:
      planes.delete(plane);
    }
  }

  // Then, let's handle all the planes that are still tracked.
  // This consists both of tracked planes that we have previously seen (may have
  // been updated), and new planes.
  detectedPlanes.forEach(plane => {
    if (planes.has(plane)) {
      // Handle previously-seen plane:

      if(plane.lastChangedTime > planes.get(plane)) {
        // Handle previously seen plane that was updated.
        // It means that one of the plane's properties is different than
        // it used to be - most likely, the polygon has changed.

        ... // Render / prepare the plane for rendering, etc.

        // Update the time when we have updated the plane:
        planes.set(plane, plane.lastChangedTime);
      } else {
        // Handle previously seen plane that was not updated in current frame.
        // Note that plane's pose relative to some other space MAY have changed.
      }
    } else {
      // Handle new plane.

      // Set the time when we have updated the plane:
      planes.set(plane, plane.lastChangedTime);
    }

    // Irrespective of whether the plane was previously seen or not,
    // & updated or not, its pose MAY have changed:
    const planePose = frame.getPose(plane.planeSpace, xrReferenceSpace);
  });

  frame.session.requestAnimationFrame(onXRFrame);
}
</pre>

</div>

Native device concepts {#native-device-concepts}
======================

Native plane detection {#native-plane-detection-section}
----------------------

<section class="non-normative">

The plane detection API provides information about flat surfaces detected in users' environment. It is assumed in this specification that user agents can rely on <dfn>native plane detection</dfn> capabilities provided by the underlying platform for their implementation of plane-detection features. Specifically, the underlying XR device should provide a way to query all planes that are tracked at a time that corresponds to the [=XRFrame/time=]of a specific {{XRFrame}}.

Moreover, it is assumed that the tracked planes, known as <dfn>native plane objects</dfn>, maintain their identity across frames - that is, given a plane object <code>P</code> returned by the underlying system at time <code>t0</code>, and a plane object <code>Q</code> returned by the underlying system at time <code>t1</code>, it is possible for the user agent to query the underlying system about whether <code>P</code> and <code>Q</code> correspond to the same logical plane object. The underlying system is also expected to provide a [=native origin=] that can be used to query the location of a pose at time <code>t</code>, although it is not guaranteed that plane pose will always be known (for example for planes that are still tracked but not localizable at a given time). In addition, the native plane object should expose a polygon describing approximate shape of the detected plane.

In addition, the underlying system should recognize native planes as native entities for the purposes of {{XRAnchor}} creation. For more information, see [[webxr-anchors-module#native-anchor]] section.

</section>

Privacy & Security Considerations {#privacy-security}
=================================

<section class="non-normative">

The plane detection API exposes information about users' physical environment. The exposed plane information (such as plane's polygon) may be limited if the user agent so chooses. Some of the ways in which the user agent can reduce the exposed information are: decreasing the level of detail of the plane's polygon in [=update plane object=] algorithm (for example by decreasing the number of vertices, or by [=rounding=] / [=quantization|quantizing=] the coordinates of the vertices), or removing the plane altogether by behaving as if the plane object was not present in <code>trackedPlanes</code> collection in [=update planes=] algorithm (this could be done for example if the detected plane is deemed to small / too detailed to be surfaced and the mechanisms to reduce details exposed on planes are not implemented by the user agent). The poses of the planes (obtainable from {{XRPlane/planeSpace}}) could also be [=quantization|quantized=].

Since concepts from plane detection API can be used in methods exposed by [[webxr-anchors-module]] specification, some of the privacy & security considerations that are relevant to WebXR Anchors Module also apply here. For details, see [[webxr-anchors-module#privacy-security]] section.

Due to how plane detection API extends WebXR Device API, the section [[webxr#security]] is also applicable to the features exposed by the WebXR Plane Detection Module.

</section>

Acknowledgements {#ack}
================

The following individuals have contributed to the design of the WebXR Plane Detection specification: