Table of Contents

Input frame data requirements for external frame data sources

To make an external frame data source work correctly, the most important and also the trickiest task is to ensure the correctness of the data. This article describes the input frame data requirements for external frame data sources.

Before you begin

Input frame data types

In Unity, an external frame data source usually needs to receive different data at two different times. According to the input time and characteristics of the external data, these two groups of data are called:

  1. camera frame data
  2. rendering frame data

Different types of external frame data sources require these two groups of data differently:

  • Image and device motion data input extension: requires both camera frame data and rendering frame data
  • Image input extension: requires only camera frame data

Camera frame data

Data requirements:

  1. timestamp
  2. raw camera image data
  3. intrinsics, including image size, focal length, and principal point. If distortion exists, the distortion model and distortion parameters are also required
  4. extrinsics, Tcw or Twc, a calibrated matrix that expresses the physical offset of the physical camera relative to the device/head pose origin
  5. tracking status
  6. device pose

Data time:

  • Middle point of the physical camera exposure

Data usage:

  • API call time: can vary according to the design of the external code. A common method used by most devices is to query during the rendering update of the 3D engine, and then decide whether to further process the data according to the timestamp of the device data
  • API call thread: the game thread of the 3D engine, or any other thread if all external APIs used are thread-safe

The API call example in Unity is as follows:

void TryInputCameraFrameData()
{
    double timestamp;

    if (timestamp == curTimestamp) { return; }
    curTimestamp = timestamp;

    PixelFormat format;
    Vector2Int size;
    Vector2Int pixelSize;
    int bufferSize;

    var bufferO = TryAcquireBuffer(bufferSize);
    if (bufferO.OnNone) { return; }
    var buffer = bufferO.Value;

    IntPtr imageData;
    buffer.tryCopyFrom(imageData, 0, 0, bufferSize);

    var historicalHeadPose = new Pose();
    MotionTrackingStatus trackingStatus = (MotionTrackingStatus)(-1);

    using (buffer)
    using (var image = Image.create(buffer, format, size.x, size.y, pixelSize.x, pixelSize.y))
    {
        HandleCameraFrameData(deviceCamera, timestamp, image, cameraParameters, historicalHeadPose, trackingStatus);
    }
}

Render frame data

Data requirements:

  1. Timestamp
  2. Tracking status
  3. Device pose

Data time:

  • The time when the frame is presented on screen. TimeWarp is not included. The device pose data at the same time will be used externally, for example by the device SDK, to set the transform of the virtual camera for rendering the current frame.
Note

TimeWarp, sometimes also called Reprojection or ATW/PTW, is a common latency-reduction technique used in VR/AR headsets. After rendering is complete, it warps the image again according to the latest head pose to compensate for head motion generated during rendering. EasyAR needs the time corresponding to the pose used to set the virtual camera at the start of rendering, not the actual screen presentation time after TimeWarp.

Data usage:

  • API call time: each rendering frame of the 3D engine
  • API call thread: the game thread of the 3D engine

An example of API calls in Unity is as follows:

private void InputRenderFrameMotionData()
{
    double timestamp = 0e-9;
    var headPose = new Pose();
    MotionTrackingStatus trackingStatus = (MotionTrackingStatus)(-1);
    HandleRenderFrameData(timestamp, headPose, trackingStatus);
}

Data requirement details

Physical camera image data:

  • Image coordinate system: data acquired when the sensor is horizontal should also be horizontal. The data should use the upper-left corner as the origin and be stored in row-major order. The image should not be flipped or inverted.
  • Image FPS: normal data at 30 or 60 fps is acceptable. If high fps has special impact, the minimum acceptable frame rate for reasonable algorithm results is 2. It is recommended to use fps higher than 2, and normally the original data frame rate can be used.
  • Image size: to obtain better computation results, the longest side should be 960 or larger. Time-consuming image scaling in the data pipeline is normally discouraged. It is recommended to use the original data directly unless the copy time of the full-size data is already unacceptably long. The image resolution must not be lower than 640*480.
  • Pixel format: with tracking quality prioritized and performance considered together, the usual preferred order is YUV > RGB > RGBA > Gray (the Y component in YUV). When using YUV data, a complete data definition is required, including data packing and padding details. Compared with single-channel images, color images provide better Mega results, but have little effect on other features.
  • Data access: data pointer or an equivalent implementation. It is best to eliminate all possible unnecessary copies in the data pipeline. In HandleRenderFrameData, EasyAR copies one copy of the data and then uses it asynchronously. After the synchronous call completes, the image data is no longer used. Pay attention to data ownership.

Timestamps:

  • All timestamps should be clock-synchronized, preferably hardware-synchronized. The data unit is seconds, but precision should reach nanoseconds or be as high as possible.

Tracking status:

  • Tracking status is defined by the device and must include the tracking lost state, where VIO is unavailable. More levels are better if available.

Device pose:

  • All poses, including the transform of the virtual camera in the 3D engine, should use the same origin.
  • All poses and extrinsics should use the same coordinate-axis system.
  • In Unity, the coordinate-axis system type of pose data should be either the Unity coordinate-axis system or the EasyAR coordinate-axis system. If the input extension is implemented by EasyAR and uses another coordinate-axis system definition, provide a clear definition of the coordinate-axis system or a method to convert it to the Unity coordinate-axis system or the EasyAR coordinate-axis system.
  • In Unity, if the Unity XR framework is used, compatibility with XROrigin.TrackingOriginMode.Device mode is sufficient.

Intrinsics:

  • All values should match the image data. Scale the intrinsics before inputting them to EasyAR if needed.
  • If the input extension is implemented by EasyAR, specify whether intrinsics change every frame, which determines whether the corresponding API should be called once or every frame.

Extrinsics:

  • Real data must be provided on headsets.
  • This is a calibration matrix expressing the physical offset of the physical camera relative to the pose origin of the device/head. If the device pose and physical camera pose are the same, it should be an identity matrix.
  • The corresponding interface for Apple Vision Pro is CameraFrame.Sample.Parameters.extrinsics. Note that its data definition differs from the data required by the interface, and EasyAR internally uses it after conversion.
  • In Unity, the coordinate-axis system type of extrinsics should be either the Unity coordinate-axis system or the EasyAR coordinate-axis system. If the input extension is implemented by EasyAR and uses another coordinate-axis system definition, provide a clear definition of the coordinate-axis system or a method to convert it to the Unity coordinate-axis system or the EasyAR coordinate-axis system.
  • Headset devices usually contain multiple coordinate systems with different definitions, including differences in coordinate-axis origins, orientations, and left- or right-handed representations. Extrinsics should be calculated under the same coordinate system. The interface data requires coordinate transformation under the same coordinate system, not a transformation matrix between two coordinate systems with different definitions.

Performance:

  • Data should be provided with optimal efficiency. In most implementations, API calls occur during rendering, so even if time-consuming operations are required at the lower layer, avoid blocking API calls or use these APIs in a reasonable way.
  • If the input extension is implemented by EasyAR, describe all time-consuming API calls.

Multi-camera:

  • At least one camera's data is required. This camera can be any RGB camera, VST camera, localization camera, and so on. On headsets, if only one camera's data is input, it is usually recommended to use an RGB camera or VST camera near the center or near the eyes.
  • Using multiple cameras can improve the EasyAR algorithm result. Camera frame data from all available cameras at a given moment should be input simultaneously at the same time point.

Multi-camera is not yet fully supported. Contact EasyAR for more details.

Next steps