Ark

The Auki Unity Ark package provides a set of functionalities for QR code corner detection, bit array scanning, pose estimation, and pose optimization within Unity applications. Such core features, originally part of Manna, are now usable from any developers in need of the sole Computer Vision functionalities implemented by Auki.

This package can be used as a standalone package.

Features

QR Code Detection

• Detect QR codes within a given texture.
• Extract information such as detected QR corners, bit arrays, size, and framestamp.

Pose Estimation

• Estimate the pose in world space for a given set of QR code corners, physical side length, and transformation matrices.
• Calculate confidence values for the estimated pose.

Pose Optimization

• Optimize the pose to find a better fit according to integrated or custom cost functions.
• Control the optimization process with customizable time limits.

Usage Examples

QR Code Detection

// Parse a texture for QR codes
Texture texture = /* provide your texture */;
var qrResults = await Ark.ParseTextureForQRs(texture);

foreach (var qrResult in qrResults)
{
    // Access QR code information
    var corners = qrResult.Corners;
    var bitArray = qrResult.BitArray;
    var size = qrResult.Size;

    // The following is a specific Auki feature: FRAMESTAMP.
    // Dynamic QR codes shown on screen also show a framestamp,
    // which encodes the currently rendered frame id.
    // This is useful for calibration and synchronization purposes.
    // When not available/detected, value is uint.MaxValue
    var framestamp = qrResult.Framestamp;

    // Process QR code information
    // ...
}

Pose Estimation

At the current moment, when providing only a projectionMatrix and worldToCameraMatrix, only ARFoundation is supported.

Camera unityArCamera; /* provide the camera */
Matrix4x4 projectionMatrix = unityArCamera.projectionMatrix;
Matrix4x4 worldToCameraMatrix = unityArCamera.worldToCameraMatrix;

// Estimate pose from QR code corners and transformation matrices
var corners = /* provide QR code corners */;
float sideLengthInM = /* provide QR code side length in meters */;
Texture texture = /* provide texture */;

var coordinateEnvironment = new CoordinateEnvironment(
    texture,
    projectionMatrix,
    worldToCameraMatrix
);

var estimationResult = Ark.EstimatePose(
    corners,
    sideLengthInM,
    coordinateEnvironment
);

if (estimationResult.Succeeded)
{
    // Access pose and confidence value
    var pose = estimationResult.Pose;
    var confidence = estimationResult.Confidence;

    // Process pose estimation result
    // ...
}

The suggested way to obtain such matrices is via our ARFoundation Integration package. This will give you the best results extracting the matrices directly from ARKit/ARCore libraries.

CameraFrameProvider tp = CameraFrameProvider.GetOrCreateComponent();
tp.OnNewFrameReady += frame =>
{
    Texture texture = frame.Texture;
    Matrix4x4 projectionMatrix = frame.ARProjectionMatrix;
    Matrix4x4 worldToCameraMatrix = frame.ARWorldToCameraMatrix;

    /* do processing */
};

A more advanced method is offered to support any pose estimation pipeline. In such case, the textureToProjectedCoordinatesMatrix (or TextureToNDC matrix) is required as an additional parameter to know more about your coordinate system.

// Estimate pose from QR code corners and transformation matrices
var corners = /* provide QR code corners */;
float sideLengthInM = /* provide QR code side length in meters */;
Matrix4x4 textureToProjectedCoordinatesMatrix = /* provide transformation matrix */;
Matrix4x4 projectionMatrix = /* provide camera projection matrix */;
Matrix4x4 worldToCameraMatrix = /* provide world-to-camera transformation matrix */;

var estimationResult = Ark.EstimatePose(
    corners,
    sideLengthInM,
    textureToProjectedCoordinatesMatrix,
    projectionMatrix,
    worldToCameraMatrix
);

in DefaultTextureToNDCMatrices class we offer helper functions to generate your own. In ARFoundation case, we internally use ARFoundationDefaultMatrix(), which would make the advanced method equivalent to the above simpler methods.

Matrix4x4 textureToProjectedCoordinatesMatrix = 
    DefaultTextureToNDCMatrices.ARFoundationDefaultMatrix(
            textureWidth,   /* image texture width in pixels */
            textureHeight,  /* image texture height in pixels */
            screenWidth,    /* phone resolution width in pixels */
            screenHeight,   /* phone resolution height in pixels */
            ScreenOrientation screenOrientation   /* current phone orientation */
    );

To generate your own, you can use the more generic version, which adds more fine tuning depending on your texture coordinates convention. More details in the code documentation.

Matrix4x4 textureToProjectedCoordinatesMatrix = 
    DefaultTextureToNDCMatrices.GenericMatrix(
            TextureCorner originOfTextureInPortraitMode, /* which corner is the origin of texture coordinates */
            bool widthIsHorizontalInPortraitMode,
            bool insensitiveToLandscapeMode,
            int textureWidth,   /* image texture width in pixels */
            int textureHeight,  /* image texture height in pixels */
            int screenWidth,    /* phone resolution width in pixels */
            int screenHeight,   /* phone resolution height in pixels */
            ScreenOrientation screenOrientation   /* current phone orientation */
        )

Pose Optimization

Given a specific measured pose, we offer a method to optimize it according to a cost function.

// Optimize pose with integrated or custom cost functions
var corners = /* provide QR code corners */;
Pose poseToOptimize = /* provide initial pose */;
float sideLengthInM = /* provide QR code side length in meters */;
var coordinateEnvironment = /* provide CoordinateEnvironment */;

var optimizedPose = Ark.OptimizePose(
    corners,
    poseToOptimize,
    sideLengthInM,
    coordinateEnvironment
);

The above uses our own cost function: it iteratively constructs multiple 3D squares from slight increments of the given pose and compare its projected corners with the given 2D corners. With enough iterations, it might converge to better estimations.

Alternatively you can provide your own cost function: given a Pose, return a cost value. A lower cost is expected to be a better pose.

// Define the pose to be optimized
Pose poseToOptimize = /* provide initial pose */;
float optimizationTimeLimitInMs = /* provide time limit for optimization in milliseconds */;
Func<Pose, float> costFunction = /* provide cost function */;

// Call the OptimizePose method
var optimizedPose = Ark.OptimizePose(
    poseToOptimize,
    costFunction,
    optimizationTimeLimitInMs
);

Notes

• Handle asynchronous tasks appropriately when using methods that return Task or awaitable results.
• Refer to the Unity documentation for further details on matrix transformations and pose calculations.