Auki::Ur::HandTracker

Ur's HandTracker provides real-time tracking of 3D hand poses from monocular RGB video in mobile phones.

Public Classes

	Name
struct	CalibrationReport Data structure used to report information about the current calibration process in the StartCalibration callback.
struct	CalibrationState Describes a CalibrationState.

Public Types

	Name
enum class	CalibrationStatus { NOT_CALIBRATED, CALIBRATING, CALIBRATED, MANUAL_CALIBRATION} Enum with the possible HandTracker calibration state values.
enum class	CalibrationStatusReport { FAILURE_NO_PLANE, FAILURE_NO_HAND, FAILURE_NO_MEASUREMENTS, CALIBRATION_AR_NOT_READY, CALIBRATION_PROGRESS, CALIBRATION_FINISHED} Enum used to report calibration state during calibration.

Public Functions

	Name
HandTracker	GetInstance() Returns the instance of the HandTracker (singleton).
void	SetARSystem(ARSession arSession, Camera arCamera, ARRaycastManager arRaycastManager) Pass along the AR state the client app is using. It is required to call this method with valid parameters before starting HandTracker.
void	SetMaterialOverride(Material material) Method to override the hand tracker debugger default material. This has to be called before calling HandTracker.Start()
void	StartCalibration(Action< CalibrationReport > progressReport, bool showUI =false) Starts the HandTracker's calibration process. A delegate can be used to get progress report from the calibration process.
CalibrationStatus	GetCalibrationStatus() Query the calibration status of HandTracker.
CalibrationState	GetCalibrationState() Returns the HandTracker's current CalibrationState. Make sure the calibration status is CALIBRATED before relying on this state.
void	SetCalibrateState(CalibrationState calibrationState) Sets the HansTracker's current CalibrationState. After this call, the calibration status will be set to MANUAL_CALIBRATION to reflect the fact that the calibration state was restored from a previous session.
bool	Start(int numHands =1, long updateIntervalInMilliseconds =0) Begins tracking hand(s).
void	Stop() Stops tracking hand(s).
void	ShowHandMesh() Displays debug GameObjects on tracked hand landmarks.
void	HideHandMesh() Hides debug GameObjects on tracked hand landmarks.
void	Update() Processes camera frames to track hands. Needs to be invoked in the game loop.
float[]	Get3DLandmarks() Gets a list of hand landmarks.
float[]	GetTranslations() Gets the global translation of the hand(s).

Public Attributes

	Name
const int	LandmarksCount Number of landmarks per hand.
const int	TranslationsCount Number of translations.
Action< float[], float[], int[], float[]>	OnUpdate Callback returning estimated landmarks, their translations, right-handedness and confidence scores.

Public Types Documentation

enum CalibrationStatus

Enumerator	Value	Description
NOT_CALIBRATED		tracker is not yet calibrated
CALIBRATING		tracker is ongoing calibration
CALIBRATED		tracker is calibrated
MANUAL_CALIBRATION		manual calibration

Enum with the possible HandTracker calibration state values.

enum CalibrationStatusReport

Enumerator	Value	Description
FAILURE_NO_PLANE		reported when no plane trackable can be found
FAILURE_NO_HAND		reported when hand is not detected
FAILURE_NO_MEASUREMENTS		reported when no measurements (or not enough measurements) where taken. This could be due to user moving the device too fast, making it impossible to take reliable measurements.
CALIBRATION_AR_NOT_READY		reported while awaiting for the user to move the mobile device to gather more information about the environment
CALIBRATION_PROGRESS		reported during the calibration process whenever enough measurements have been taken
CALIBRATION_FINISHED		reported when there are sample measurements for all depth intervals and a calibration

Enum used to report calibration state during calibration.

FAILURE_NO_PLANE is reported when no plane trackable can be found. FAILURE_NO_HAND is reported when hand is not detected. FAILURE_NO_MEASUREMENTS is reported when no measurements (or not enough measurements) where taken. This could be due to user moving the device too fast, making it impossible to take reliable measurements. CALIBRATION_AR_NOT_READY is reported while awaiting for the user to move the mobile device to gather more information about the environment CALIBRATION_PROGRESS is reported during the calibration process whenever enough measurements have been taken from one of the required depth interval. CALIBRATION_FINISHED is reported when there are sample measurements for all depth intervals and a calibration value was found.

Public Functions Documentation

function GetInstance

static HandTracker GetInstance()

Returns the instance of the HandTracker (singleton).

Return: HandTracker object

function SetARSystem

void SetARSystem(
    ARSession arSession,
    Camera arCamera,
    ARRaycastManager arRaycastManager
)

Pass along the AR state the client app is using. It is required to call this method with valid parameters before starting HandTracker.

Parameters:

• arSession The ARSession in use by the client app.
• arCamera The Camera attached to the ARSession
• arRaycastManager The ARRaycastManager in use by the client app.

function SetMaterialOverride

void SetMaterialOverride(
    Material material
)

Method to override the hand tracker debugger default material. This has to be called before calling HandTracker.Start()

Parameters:

• material

function StartCalibration

void StartCalibration(
    Action< CalibrationReport > progressReport,
    bool showUI =false
)

Starts the HandTracker's calibration process. A delegate can be used to get progress report from the calibration process.

Parameters:

• progressReport Delegate to be called with calibration progress report.
• showUI If true, a debug internal UI will be displayed to drive the calibration process. Default is false.

function GetCalibrationStatus

CalibrationStatus GetCalibrationStatus()

Query the calibration status of HandTracker.

Return: A CalibrationStatus

function GetCalibrationState

CalibrationState GetCalibrationState()

Returns the HandTracker's current CalibrationState. Make sure the calibration status is CALIBRATED before relying on this state.

Return: A CalibrationState

function SetCalibrateState

void SetCalibrateState(
    CalibrationState calibrationState
)

Sets the HansTracker's current CalibrationState. After this call, the calibration status will be set to MANUAL_CALIBRATION to reflect the fact that the calibration state was restored from a previous session.

function Start

bool Start(
    int numHands =1,
    long updateIntervalInMilliseconds =0
)

Begins tracking hand(s).

Parameters:

• numHands Number hands to track
• updateIntervalInMilliseconds Tracking update interval

Return: true if tracking starts successfully, false if it doesn't

function Stop

void Stop()

Stops tracking hand(s).

function ShowHandMesh

void ShowHandMesh()

Displays debug GameObjects on tracked hand landmarks.

function HideHandMesh

void HideHandMesh()

Hides debug GameObjects on tracked hand landmarks.

function Update

void Update()

Processes camera frames to track hands. Needs to be invoked in the game loop.

function Get3DLandmarks

float[] Get3DLandmarks()

Gets a list of hand landmarks.

Return: An array of floats with each 3 representing the x, y and z-components of a hand landmark

function GetTranslations

float[] GetTranslations()

Gets the global translation of the hand(s).

Return: An array of floats with each 3 representing the x, y and z-components of a hand translation

Public Attributes Documentation

variable LandmarksCount

static const int LandmarksCount = 21;

Number of landmarks per hand.

A landmark is a 3D vector that holds the local translation for each hand bone joint.

variable TranslationsCount

static const int TranslationsCount = 1;

Number of translations.

For each hand there is a translation, an additional 3D vector (x, y, z), which contains the global translation of the hand in camera space which all hand landmarks should be displaced with to get the landmarks in AR camera space.

variable OnUpdate

Action< float[], float[], int[], float[]> OnUpdate;

Callback returning estimated landmarks, their translations, right-handedness and confidence scores.

The landmark and translation arrays contain consecutive floats which represent the x, y and z-components of a 3D vector.