Table of Contents

Unity camera in AR scenes

Unity AR experiences rely on the camera. This article explains the role of the camera in AR scenes and how sessions control camera properties to ensure the correct AR experience.

Before you begin

The camera's role in AR scenes

In Unity, the camera is used to show the game world to the player, but in AR scenes its role is even more important. It not only renders virtual content, but also needs to align with the real world to ensure virtual objects are correctly overlaid on the real scene.

This video shows a simple AR scene. The left side is the Scene view and the right side is the Game view. The video was recorded in Unity Editor Play mode using simulation data. The content in the Game view is the same as what the user sees on the phone in the real world.

In this video, the camera representing the user (camera icon) moves according to the user's movement in the real world. The white cone shows the camera's position and orientation over the past period. In the Game view, the camera not only shows the virtual content from the Scene view, but also overlays the real-world image at the bottom of the virtual content. This is the typical behavior of a camera in an AR scene.

To ensure virtual objects are correctly overlaid on the real scene, some camera properties need to be adjusted according to the runtime state of AR. These properties include:

  • the camera transform (position and orientation)
  • the camera's field of view (FOV), aspect ratio, and projection matrix
  • the camera culling settings (<xref:UnityEngine.GL.invertCulling?displayProperty=nameWithType>)
Warning

When developing an application, modifying these camera properties on the session camera is not supported, because it may cause virtual content to align incorrectly with the real world and affect the user experience. Even if these properties are modified by some means, the AR system will overwrite those changes during runtime, or unexpected behavior may occur because rendered data and computed data are inconsistent.

Depending on the object that controls these properties, the camera used by the session can be divided into two types: cameras controlled by the session and cameras not controlled by the session.

Cameras controlled by the session

If the session camera does not belong to any external system, such as a headset or AR Foundation, the session automatically controls the above camera properties to ensure the camera is correctly aligned with the real world.

transform

The camera's transform (position and orientation) is adjusted by the session according to the runtime state of the AR feature. In general, the session updates the camera position and orientation based on motion tracking data and/or target tracking data to ensure the content seen by the user matches the real world.

In Unity, the central reference point for all AR tracking is called the session origin, and the rule that determines this origin during session runtime is called the center mode. The camera transform behaves differently in different center modes:

  • In <xref:u:easyar.ARSession.ARCenterMode.Camera> center mode, the camera can move freely.

    In general, <xref:u:easyar.ARSession.ARCenterMode.Camera> mode is rarely used by applications.

  • In other center modes, such as <xref:u:easyar.ARSession.ARCenterMode.FirstTarget>, the camera cannot move freely.

    <xref:u:easyar.ARSession.ARCenterMode.FirstTarget> is the mode used by most AR applications.

Warning

The scale of the camera transform should always remain (1, 1, 1). Modifying the camera scale may cause unexpected behavior.

Projection matrix

The camera's projection matrix is updated every frame by the session according to the physical camera intrinsics to ensure virtual content is correctly overlaid on the real scene.

Culling settings

The camera culling settings (<xref:UnityEngine.GL.invertCulling?displayProperty=nameWithType>) are adjusted according to the session's mirror settings to ensure virtual content is rendered correctly in the real scene.

When the setting corresponding to the current camera in <xref:u:easyar.ARSession.HorizontalFlip> is <xref:u:easyar.ARSession.ARHorizontalFlipMode.World>, <xref:UnityEngine.GL.invertCulling?displayProperty=nameWithType> is set to true. This is the default configuration for the front camera.

AR background video stream

In AR scenes, the camera usually renders the video stream from the physical camera as the background to enhance immersion. The session automatically handles video stream acquisition and rendering, and ensures the video stream is correctly aligned with the virtual content.

Cameras not controlled by the session

When using a headset or AR Foundation, or when the implementation specifies <xref:u:easyar.FrameSource.IsCameraUnderControl> as false for <xref:u:easyar.FrameSource>, the session does not control the camera properties above. Instead, they are controlled by the external system.

Warning

Although the session does not control the camera properties in this case, they are still controlled by a third-party system such as a headset SDK or AR Foundation, so modifying these properties during development is still unsupported.

Considerations when copying cameras

Sometimes you may need to copy the parameters of the session camera to another camera. In that case, pay extra attention to the following two points:

  • Property acquisition time: when using a controlled camera, refer to Getting session output to obtain these parameters at the correct time; when using a camera not controlled by the session, refer to the third-party system's documentation to obtain them at the correct time.
  • The camera's field of view (FOV), aspect ratio, and projection matrix: use <xref:UnityEngine.Camera.projectionMatrix?displayProperty=nameWithType> to get the camera projection matrix and copy it to the other camera. <xref:UnityEngine.Camera.fieldOfView?displayProperty=nameWithType> and <xref:UnityEngine.Camera.aspect?displayProperty=nameWithType> are mathematically part of the projection matrix, so using only <xref:UnityEngine.Camera.fieldOfView?displayProperty=nameWithType> and <xref:UnityEngine.Camera.aspect?displayProperty=nameWithType> is not sufficient.

Next steps