Use EasyAR in a 3D engine

To use EasyAR in a 3D engine, you need to render the camera image and virtual objects. The rendered virtual objects must align with the camera image. When rendering the camera image, some parameters during image generation and display may not match. For example, the physical camera position, orientation, frame size, aspect ratio, and other parameters may differ from the display image, and these need to be considered during rendering. If you need to integrate EasyAR into an unsupported 3D engine, pay special attention to the following details.

Cropping camera image boundary padding

Image cropping, transposition, and encoding all require considerable computation. To reduce computation and latency, relatively raw formats are usually used. For easier video encoding, images output by physical cameras are often aligned to grids such as 8x8, 16x16, 32x32, or 64x64. For example, when a 1920x1080 resolution is selected on some phones, the output image may become 1920x1088 because 1080 is not a multiple of 64.

This requires removing the extra padding during rendering. There are multiple possible approaches. One is to specify the width when uploading the image to video memory, for example using glPixelStorei(GL_PACK_ROW_LENGTH, ...) in OpenGL. Another is to manually calculate UV coordinates in the fragment shader and truncate the excess part when sampling from the image.

Render following screen rotation

On mobile phones, the image recorded by the physical camera is usually fixed relative to the device body and does not change with the screen display orientation. However, changes in the phone body orientation affect how we define the up, down, left, and right directions of the image. During rendering, the current screen display orientation also affects the direction of the displayed image.

Usually during rendering, you need to determine the rotation angle of the camera image relative to the screen display orientation.

Let \(\theta_{screen}\) represent the radians by which the screen image rotates clockwise relative to the natural screen orientation, \(\theta_{phycam}\) represent the radians by which the physical camera image must rotate clockwise to display correctly on a screen in the natural orientation, and \(\theta\) represent the radians by which the physical camera image must rotate clockwise to display on the current screen.

For the rear camera:

\[ \theta = \theta_{phycam} - \theta_{screen} \]

For example, on an Android phone, when the phone is used in the natural orientation, \(\theta_{screen} = 0, \theta_{phycam} = \frac{\pi}{2}\), so \(\theta = \frac{\pi}{2}\).

For the front camera, if the image is flipped horizontally after rotation is completed:

\[ \theta = \theta_{phycam} + \theta_{screen} \]

Note

When the screen image rotates, \(\theta\) must be recalculated immediately on the first frame after the rotation occurs; otherwise, the screen image direction may be temporarily incorrect.

Rendering the camera background and virtual objects

When rendering virtual objects on a mobile phone, the virtual objects must align with the camera image. This requires placing both the rendering camera and objects in a virtual space that exactly corresponds to the real space, and rendering with the same field of view and aspect ratio as the physical camera. The perspective projection transforms applied to the camera image and virtual objects are almost identical. The only difference is that most of the perspective projection transform of the camera image happens inside the physical camera, while the perspective projection transform of virtual objects is entirely a computational process.

The following uses OpenGL conventions. If other conventions are used, corresponding coordinate-axis mapping is required. Assume the coordinate axes of the camera coordinate system are defined as follows: the x-axis points right, the y-axis points up, and the z-axis points outward from the screen. The coordinate axes of the clip coordinate system are defined as follows: the x-axis points right, the y-axis points up, the z-axis points outward from the screen, and the w-axis is a virtual axis.

At this point, the perspective projection transform matrix required to render the camera image is as follows:

\[ P_i=\left( \begin{array}{cccc} (-1)^{\text{flip}} & \phantom{0} & \phantom{0} & \phantom{0} \\ \phantom{0} & 1 & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & 1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right)\left( \begin{array}{cccc} \cos (-\theta ) & -\sin (-\theta ) & \phantom{0} & \phantom{0} \\ \sin (-\theta ) & \cos (-\theta ) & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & 1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right)\left( \begin{array}{cccc} s_x & \phantom{0} & \phantom{0} & \phantom{0} \\ \phantom{0} & s_y & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & 1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right) \]

Here, flip indicates whether the image is horizontally flipped. Its value is 1 when flipped and 0 when not flipped. \(\theta\) is the clockwise image rotation angle, in radians. \(s_x\) and \(s_y\) are scale factors used for proportional scaling or proportional filling, and they vary with \(\theta\). This transform matrix first scales the camera image, then rotates it, and finally flips it. During rendering, use a rectangle that fills the screen. For example, in OpenGL, the rectangle vertices can be placed at \((-1, -1, 0)\), \((1, -1, 0)\), \((1, 1, 0)\), and \((-1, 1, 0)\), with UV coordinates set at the corresponding four corners, and then rendered with this perspective projection matrix.

The perspective projection matrix required to render virtual objects is as follows:

\[ P=P_i\left( \begin{array}{cccc} 1 & \phantom{0} & \phantom{0} & \phantom{0} \\ \phantom{0} & 1 & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & -\frac{f+n}{f-n} & -\frac{2 f n}{f-n} \\ \phantom{0} & \phantom{0} & -1 & \phantom{0} \\ \end{array} \right)\left( \begin{array}{cccc} \frac{2}{w} & \phantom{0} & \phantom{0} & \phantom{0} \\ \phantom{0} & \frac{2}{h} & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & 1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right)\left( \begin{array}{cccc} 1 & \phantom{0} & \phantom{0} & \phantom{0} \\ \phantom{0} & -1 & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & -1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right)\left( \begin{array}{cccc} f_x & \phantom{0} & c_x & \phantom{0} \\ \phantom{0} & f_y & c_y & \phantom{0} \\ \phantom{0} & \phantom{0} & 1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right)\left( \begin{array}{cccc} 1 & \phantom{0} & \phantom{0} & \phantom{0} \\ \phantom{0} & -1 & \phantom{0} & \phantom{0} \\ \phantom{0} & \phantom{0} & -1 & \phantom{0} \\ \phantom{0} & \phantom{0} & \phantom{0} & 1 \\ \end{array} \right) \]

Here, \(n\) and \(f\) are the near and far clipping parameters commonly used in 3D rendering perspective projection matrices. \(w\) and \(h\) are the pixel width and height of the camera image. \(f_x\), \(f_y\), \(c_x\), and \(c_y\) are common intrinsic parameters in the camera model, where \(f_x\) and \(f_y\) are pixel focal lengths, and \(c_x\) and \(c_y\) are the pixel positions of the principal point. This projection matrix performs the following transforms in order: the perspective projection transform of the camera intrinsics (because the y- and z-axis directions of the image coordinate system in OpenCV are opposite to the OpenGL camera coordinate system, two coordinate system transforms are performed), the transform from the image pixel coordinate system to the image rectangle coordinate system, the near and far clipping transform, and the perspective projection transform used when rendering the camera image.

After simplification, we get:

\[ P=P_i\left( \begin{array}{cccc} \frac{2 f_x}{w} & \phantom{0} & 1-\frac{2 c_x}{w} & \phantom{0} \\ \phantom{0} & \frac{2 f_y}{h} & -1+\frac{2 c_y}{h} & \phantom{0} \\ \phantom{0} & \phantom{0} & -\frac{f+n}{f-n} & -\frac{2 f n}{f-n} \\ \phantom{0} & \phantom{0} & -1 & \phantom{0} \\ \end{array} \right) \]

From the process above, rendering usually needs to be performed in two passes: one pass renders the camera image, and one pass renders the virtual objects, with the virtual objects overlaid on top of the camera image.

Some 3D engines represent the perspective projection matrix with parameters such as horizontal field of view and aspect ratio. If rotation and flipping are not considered and principal point offset is ignored, these can be calculated, where the horizontal field of view is \(\alpha=2 arctan{\frac{w}{2 f_x}}\) and the aspect ratio is \(r=\frac{w}{h}\).

Note that camera distortion is not considered in this process, because the camera distortion of most current mobile phones is very slight.