Create a simple shared AR experience

This section describes how to use ConjureKit and Manna module to create a shared AR experience from scratch in Unity.

Before you begin

Before getting started, set up your development environment by following the steps in the Quickstart.

Preparing the project with ARFoundation

1. Configure the project for XR development. Go to Edit -> Project Settings -> XR Plug-In Management
   • -> iOS and enable the ARKit checkbox for iOS devices.
   • -> Android and enable the ARCore checkbox for Android devices.
2. Configure build parameters. Go to Project Settings -> Player
   • -> iOS -> Other Settings -> Camera Usage Description and write a camera description, e.g., "AR requires a camera".
     • On Android, this permission is automatically added to manifest on build. Android will ask for Camera permissions when the app will first run.
   • -> Android -> Other Settings and have
     • Auto Graphics API disabled and OpenGLES3 at the top of the list
     • Scripting backend set to IL2CPP and both ARMv7 and ARM64 selected in Target Architectures
3. Go to Package Manager -> Unity Registry, and Install ARFoundation.

caution

We suggest to check which ARFoundation and ARKit/ARCore package versions are installed by Package Manager, as they might not be equal. This is not automatically handled when installing a "Preview" version and mismatch can happen.

note

ARFoundation versions below 4.2.6 do not support iOS 16.

4. Start with an empty scene with no GameObjects.
5. Create a new AR Session Origin by selecting GameObject -> XR -> AR Session Origin.
6. Add a new ARTrackedImageManager component to the AR Session Origin GameObject.
7. Create a new ReferenceImageLibrary by selecting Assets -> Create -> XR -> Reference Image Library. and drag it to the “serialized library” field in the ARTrackedImageManager component.
8. Set "Max Number Of Moving Images" field in the ARTrackedImageManager component to 1.
9. Create a new AR Session by selecting GameObject -> XR -> AR Session.

Initializing ConjureKit

1. Install the ConjureKit package.
2. Create a new MonoBehaviour script and attach it to an empty game object in the scene.
3. Import ConjureKit.

using Auki.ConjureKit;

4. Create a private IConjureKit variable.

private IConjureKit _conjureKit;

5. Create a public Camera variable.

public Camera arCamera;

6. Drag the AR Camera to the arCamera field on the GameObject you created;

7. Initialize ConjureKit with the app key and secret from the posemesh console (cf. the Quickstart guide)

caution

Never share your app secret with anyone.

8. Call _conjureKit.Connect() to connect to a Hagall server with the lowest latency and create a session.

using Auki.ConjureKit;

public class ConjureKitDemo : MonoBehaviour
{
    public Camera arCamera;

    private IConjureKit _conjureKit;

    private void Start()
    {
        _conjureKit = new ConjureKit(
            arCamera.transform,
            "YOUR_APP_KEY",
            "YOUR_APP_SECRET");


        _conjureKit.Connect();
    }
}

9. Declare two new Text fields to display the session ID and session state on screen.

[SerializeField] private Text sessionState;
[SerializeField] private Text sessionID;

10. Register a callback to the _conjureKit.OnStateChanged event that fires every time our session state changes.

_conjureKit.OnStateChanged += state =>
{
    sessionState.text = state.ToString();
};

11. Register callback to _conjureKit.OnJoined and _conjureKit.OnLeft which are triggered when we join and leave a session.

_conjureKit.OnJoined += session =>
{
    sessionID.text = session.Id;
};

_conjureKit.OnLeft += session =>
{
    sessionID.text = "";
};

12. Create two Text elements and drag and drop them to the fields you declared in step 8.

13. Click Play in Unity Editor. Now you'll see the state of your connection logged in the console and session id and state on the screen.

Adding an entity

Follow these steps to add an Entity to the session and instantiate a Primitive Cube that would appear in the same location in AR for all participants in the session:

1. Create a method, CreateCubeEntity, that adds an Entity to the session

public void CreateCubeEntity()
{
    if (_conjureKit.GetState() != State.Calibrated)
        return;

    Vector3 position = arCamera.transform.position + arCamera.transform.forward * 0.5f;
    Quaternion rotation = Quaternion.Euler(0, arCamera.transform.eulerAngles.y, 0);

    Pose entityPos = new Pose(position, rotation);

    _conjureKit.GetSession().AddEntity(
        entityPos,
        onComplete: entity => CreateCube(entity),
        onError: error => Debug.Log(error));
}

Calling AddEntity adds a new Entity to the Session and invokes the onComplete callback, CreateCube, with the newly created entity.

_conjureKit.GetSession().AddEntity(
        entityPos,
        onComplete: entity => CreateCube(entity),
        onError: error => Debug.Log(error));

2. We need something to visualize the Entities. Declare a public GameObject variable to reference a cube prefab. Implement the CreateCube method that instantiates a cube with a Pose.

[SerializeField] private GameObject cube;

private void CreateCube(Entity entity)
{
    if (entity.Flag == EntityFlag.EntityFlagParticipantEntity) return;

    var pose = _conjureKit.GetSession().GetEntityPose(entity);
    Instantiate(cube, pose.position, pose.rotation);
}

We first chack that the entity is not the special participant entity that is created automatically when a participant joins a session.

3. Declare a Button variable to reference the button that will invoke CreateCubeEntity method.

[SerializeField] private Button spawnButton;

We want this button to be interactable only when the session state is calibrated. Add a ToggleControlsState method and invoke it in _conjureKit.OnStateChanged callback.

_conjureKit.OnStateChanged += state =>
{
    sessionState.text = state.ToString();
    ToggleControlsState(state == State.Calibrated);
};

4. Create a primitive cube by selecting GameObject -> 3D Object -> Cube. Change the scale of the cube to 0.1 so it appears as a 10cm cube in AR. Drag and drop it into the Assets folder in the Project window to create a prefab and delete the cube from the scene. Drag the cube prefab to the field you declared in step 2.

5. Add a Button to the scene, configure the on click callback to invoke CreateCubeEntity method. Drag the button game object to the field you declared in step 3.

If you run the project now you should see the cube in front of the camera.

Instant calibration

Manna module makes it easy to join a session and get instantly calibrated into a shared coordinate system by scanning a Lighthouse (a QR code in this case). Follow these steps to install and use Manna:

1. Open the Package Manager and install the Manna module.

   tip

   If you are experiencing issues with Unity editor when installing a new package, try restarting Unity.

2. Import Manna

using Auki.ConjureKit.Manna;

2. Create a private Manna variable and initialize it right after ConjureKit

private Manna _manna;

_manna = new Manna(_conjureKit);

3. Declare a Button variable that will enable and disable the QR code and a bool variable to save the QR code visibility state.

[SerializeField] bool qrCodeBool;
[SerializeField] Button qrCodeButton;

4. Toggle the QR code button interactable state in ToggleControlsState method.

private void ToggleControlsState(bool interactable)
{
    if (spawnButton) spawnButton.interactable = interactable;
    if (qrCodeButton) qrCodeButton.interactable = interactable;
}

5. Add a ToggleLighthouse method and use _manna.SetLighthouseVisible(true) to show and hide the Lighthouse (QR code).

public void ToggleLighthouse()
{
    qrCodeBool = !qrCodeBool;
    _manna.SetLighthouseVisible(qrCodeBool);
}

6. Feed Manna with AR camera video frames acquired from ARCameraManager to recognize QR codes

note

Manna is independent of any specific AR SDK, like ARFoundation. You need to manually choose how to Feed Manna With Video Frames. An example of how to do it in conjunction with ARFoundation can be found in below and in the Manna demo sample.

private void Update()
{
    FeedMannaWithVideoFrames();
}

private void FeedMannaWithVideoFrames()
{
    var imageAcquired = arCameraManager.TryAcquireLatestCpuImage(out var cpuImage);
    if (!imageAcquired)
    {
        AukiDebug.LogInfo("Couldn't acquire CPU image");
        return;
    }

    if (_videoTexture == null) _videoTexture = new Texture2D(cpuImage.width, cpuImage.height, TextureFormat.R8, false);

    var conversionParams = new XRCpuImage.ConversionParams(cpuImage, TextureFormat.R8);
    cpuImage.ConvertAsync(
        conversionParams,
        (status, @params, buffer) =>
        {
            _videoTexture.SetPixelData(buffer, 0, 0);
            _videoTexture.Apply();
            cpuImage.Dispose();

            _manna.ProcessVideoFrameTexture(
                _videoTexture,
                arCamera.projectionMatrix,
                arCamera.worldToCameraMatrix
            );
        }
    );
}

7. Add a Button to the scene, configure the on click callback to invoke ToggleLighthouse method. Drag the button game object to the field you declared in step 5.

8. Subscribe to OnEntityAdded event to create cubes when other participants add entities in the session.

_conjureKit.OnEntityAdded += CreateCube;

Build to XCode

In File -> Build settings, make sure that you have added open scenes before building to Xcode.

Another Device B running the same app as Device A can now join Device A's Session by scanning the Lighthouse (QR code) of Device A.

Complete code

using UnityEngine;
using Auki.ConjureKit;
using Auki.ConjureKit.Manna;
using Auki.Util;
using UnityEngine.UI;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

public class ConjureKitManager : MonoBehaviour
{
    [SerializeField] private Camera arCamera;
    
    [SerializeField] private Text sessionState;
    [SerializeField] private Text sessionID;
    
    [SerializeField] private GameObject cube;
    [SerializeField] private Button spawnButton;
    
    [SerializeField] bool qrCodeBool;
    [SerializeField] Button qrCodeButton;
    
    private IConjureKit _conjureKit;
    private Manna _manna;
    
    private ARCameraManager arCameraManager;
    private Texture2D _videoTexture;
    
    void Start()
    {
        arCameraManager = arCamera.GetComponent<ARCameraManager>();
        
        _conjureKit = new ConjureKit(
            arCamera.transform,
            "YOUR_APP_KEY",
            "YOUR_APP_SECRET");
    
         _manna = new Manna(_conjureKit);
        
         _conjureKit.OnStateChanged += state =>
         {
             sessionState.text = state.ToString();
             ToggleControlsState(state == State.Calibrated);
         };
    
         _conjureKit.OnJoined += session =>
         {
             sessionID.text = session.Id.ToString();
         };
    
         _conjureKit.OnLeft += session =>
         {
             sessionID.text = "";
         };
    
        _conjureKit.OnEntityAdded += CreateCube;    
        _conjureKit.Connect();
    }

    private void Update()
    {
        FeedMannaWithVideoFrames();
    }
    
    private void FeedMannaWithVideoFrames()
    {
        var imageAcquired = arCameraManager.TryAcquireLatestCpuImage(out var cpuImage);
        if (!imageAcquired)
        {
            AukiDebug.LogInfo("Couldn't acquire CPU image");
            return;
        }

        if (_videoTexture == null) _videoTexture = new Texture2D(cpuImage.width, cpuImage.height, TextureFormat.R8, false);

        var conversionParams = new XRCpuImage.ConversionParams(cpuImage, TextureFormat.R8);
        cpuImage.ConvertAsync(
            conversionParams,
            (status, @params, buffer) =>
            {
                _videoTexture.SetPixelData(buffer, 0, 0);
                _videoTexture.Apply();
                cpuImage.Dispose();

                _manna.ProcessVideoFrameTexture(
                    _videoTexture,
                    arCamera.projectionMatrix,
                    arCamera.worldToCameraMatrix
                );
            }
        );
    }

    private void ToggleControlsState(bool interactable)
    {
        if (spawnButton) spawnButton.interactable = interactable;
        if (qrCodeButton) qrCodeButton.interactable = interactable;
    }
    
    public void ToggleLighthouse()
    {
        qrCodeBool = !qrCodeBool;
        _manna.SetLighthouseVisible(qrCodeBool);
    }
    
    public void CreateCubeEntity()
    {
        if (_conjureKit.GetState() != State.Calibrated)
            return;
    
        Vector3 position = arCamera.transform.position + arCamera.transform.forward * 0.5f;
        Quaternion rotation = Quaternion.Euler(0, arCamera.transform.eulerAngles.y, 0);
    
        Pose entityPos = new Pose(position, rotation);
    
        _conjureKit.GetSession().AddEntity(
            entityPos,
            onComplete: entity => CreateCube(entity),
            onError: error => Debug.Log(error));
    }
    
    private void CreateCube(Entity entity)
    {
        if (entity.Flag == EntityFlag.EntityFlagParticipantEntity) return;
    
        var pose = _conjureKit.GetSession().GetEntityPose(entity);
        Instantiate(cube, pose.position, pose.rotation);
    }
}

The full code for this tutorial can be found on GitHub on the tutorial/simple-shared-experience branch.

The complete project with all parts and the latest packages is on the master branch of the same repo.