Skip to main content

Entity Component System

This section describes how to use ConjureKit ECS (Entity, Component, System) to make the shared AR experience project more interactive.

Before you begin​

Before getting started, complete the following steps:

  1. Set up your development environment by following the steps in the Quickstart.
  2. Finish the Simple Shared AR Experience and the Handtracker tutorials or clone the project from GitHub.

Introduction​

ConjureKit implements Entity Component System (ECS) architecture to share data between participants and entities. An ECS comprises entities composed of components of data, with systems that operate on entities' components.

Glossary​

Entity: A general-purpose object that is uniquely identifiable. It could represent a participant or a 3d asset in a session, a player or enemy character, etc. Entities are identifiers with no inherent behavior or data, simply an integer (uint) in ConjureKit.

Component: A component represents some aspect we want to attach to Entities. For example, ConjureKit Entities need to have a position and rotation. The Pose Component in ConjureKit holds the position and rotation information of an entity. Other examples of component types can be a “Health” or a “Speed” component with float data to represent the current speed or health of an Entity, a "Character” or an “Environment” component that can store data about visuals. The data stored in a component is pure data (byte[] in ConjureKit) and contains no behavior.

System: Systems are processes that operate on entities with certain components. Continuing with the Pose example, there is a PoseSystem whose only responsibility is handling entity position and rotation updates. Systems can add, remove or modify Components during runtime.

Updating the shared cube color​

After finishing the previous tutorials, you should have a project where users can join a shared AR session, create a primitive cube, and change the color of the cube by touching it with their hand, using Ur (hand tracker module). All the participants can see the cubes that other people created but will only see the color updates on the cubes touched by them. To change that, we will create a new component that will store the color data in the session and a system that will add and update the components and keep the color data locally. When a participant touches a cube, the color will be updated locally on that participant's device and a message will be broadcast to all other participants with the new color data.

Adding a Color Component and System​

Let's start by creating a color component and a color system.

Create a new ColorSystem C# class that inherits from SystemBase. ConjureKit will invoke the abstract methods defined in SystemBase.

public class ColorSystem : SystemBase
{   
    public ColorSystem(Session session) : base(session)
    {
    }
}

Declare a new component name.

// The unique name of the component
private const string COLOR_COMPONENT_NAME = "color";

And override the abstract method of SystemBase - GetComponentTypeNames to return the color component. This method is called by ConjureKit during initialization. The system is notified when components of the type reported by this method are updated or removed.

public override string[] GetComponentTypeNames()
{
    return new[] {COLOR_COMPONENT_NAME};
}

To get a list of all components that other participants have updated, let's create an override method for The SystemBase Update method, ConjureKit invokes this method when the Hagall server broadcasts a component update. Using this method, we will update a local map of component data and trigger an event for each update.

So let's declare the map of component data as a Dictionary :

// Local Color component data map
private readonly IDictionary<uint, Color> _entityColorDataMap = new Dictionary<uint, Color>();

And the event we want to trigger on every update.

/// Triggered when a component data is updated by another participant
public event Action<uint, Color> OnColorComponentUpdated;

And finally, let's override the Update method.

public override void Update(IReadOnlyList<(EntityComponent component, bool localChange)> updated)
{
    foreach (var (entityComponent, localChange) in updated)
    {
        // Update the local data and notify about the update
        _entityColorDataMap[entityComponent.EntityId] = ByteArrayToColor(entityComponent.Data);
        OnColorComponentUpdated?.Invoke(entityComponent.EntityId, _entityColorDataMap[entityComponent.EntityId]);
    }

}

Get and Set the Color​

The component data is a byte[], and it's up to the developer to decide what data to store and how. Create two methods to convert Color to byte[] and vice versa;

// Convert Color32 to byte array
private byte[] ColorToByteArray(Color32 color)
{
    byte[] colorBytes = new byte[4];
    colorBytes[0] = color.r;
    colorBytes[1] = color.g;
    colorBytes[2] = color.b;
    colorBytes[3] = color.a;

    return colorBytes;
}

// Convert byte array to Color32 
public Color32 ByteArrayToColor(byte[] bytes)
{
    if (bytes.Length < 4)
    {
        Debug.LogError("Byte array must have at least 4 elements (R, G, B, A).");
        return Color.clear;
    }

    byte r = bytes[0];
    byte g = bytes[1];
    byte b = bytes[2];
    byte a = bytes[3];

    Color32 color = new Color32(r, g, b, a);
    return color;
}

Now create a SetColor method that will update the component with the new color locally and broadcast the update to all the other participants. We'll need to first verify that an entity with the given id exists. If the entity exists but doesn't yet have the color component, add a new one.

public bool SetColor(uint entityId, Color color)
{
    // Check if the entity with the given id exists
    var entity = _session.GetEntity(entityId);
    if (entity == null) return false;
    
    // Store the data locally  
    _entityColorDataMap[entityId] = color;

    // If the entity doesn't already have Color component add one
    var component = _session.GetEntityComponent(entityId, COLOR_COMPONENT_NAME);
    if (component == null)
    {
        _session.AddComponent(
            COLOR_COMPONENT_NAME,
            entityId,
            ColorToByteArray(color),
            () => {}, 
            error => Debug.LogError(error) 
        );

        return true;
    }
    else
    {
        return _session.UpdateComponent(
            COLOR_COMPONENT_NAME,
            entityId, 
            ColorToByteArray(color)
        );
    }
}

Create a GetColor method that will return the locally stored value or a default one if the component or the entity doesn't exist. 

public Color GetColor(uint entityId)
{
    if (_session.GetEntity(entityId) == null || !_entityColorDataMap.ContainsKey(entityId))
        return Color.clear;
    
    return _entityColorDataMap[entityId];
}

Override the Delete method that will be called by ConjureKit when other participants remove a component from an entity and a DeleteColor method to delete the color locally.

public override void Delete(IReadOnlyList<(EntityComponent component, bool localChange)> deleted)
{
    foreach (var (entityComponent, localChange) in deleted)
    {
        var entity = _session.GetEntity(entityComponent.EntityId);
        if (entity == null) continue;

        _entityColorDataMap.Remove(entity.Id);
    }
}

In this tutorial, we are not going to implement this functionality, but it is necessary to override the Delete method. You can later implement a public DeleteColor method that can be called to remove a color component from an entity.

public void DeleteColor(uint entityId)
{
    _session.DeleteComponent(COLOR_COMPONENT_NAME, entityId, () =>
    {
        _entityColorDataMap.Remove(entityId);
    });
}

Initialize the system and update the color for everyone​

In the ConjureKitManager class, declare a ColorSystem and a Dictionary to store the cubes.

private ColorSystem _colorSystem;
private Dictionary<uint, Renderer> _cubes = new Dictionary<uint, Renderer>();

In Conjurekit.Onjoined callback, register the color system to the joining participant

_conjureKit.OnJoined += session =>
{
    sessionID.text = session.Id.ToString();
    
    _colorSystem = new ColorSystem(session);
    session.RegisterSystem(_colorSystem, () => Debug.Log("System registered in session"));
    _colorSystem.OnColorComponentUpdated += OnColorComponentUpdated;
};

Create a method that will change the cube's color when its component is updated.

private void OnColorComponentUpdated(uint entityId, Color color)
{
    _cubes[entityId].material.color = color;
}

Simplify the TouchableByHand class to stop updating the color locally and instead trigger an event when the cube is touched.

public class TouchableByHand : MonoBehaviour
{
    public event Action OnTouched;

    private void OnTriggerEnter(Collider other)
    {
        if (other.CompareTag("hand"))
        {
            OnTouched?.Invoke();
        }
    }
}

Update the CreateCubeEntity and the CreateCube methods to initialize a cube with white color and update with random color when touched. 

public void CreateCubeEntity()
{
    if (_conjureKit.GetState() != State.Calibrated)
        return;

    Vector3 position = arCamera.transform.position + arCamera.transform.forward * 0.5f;
    Quaternion rotation = Quaternion.Euler(0, arCamera.transform.eulerAngles.y, 0);

    Pose entityPos = new Pose(position, rotation);

    _conjureKit.GetSession().AddEntity(
        entityPos,
        onComplete: entity =>
        {
            // Initialize with white color
            _colorSystem.SetColor(entity.Id, Color.white);
            
            CreateCube(entity);
        },
        onError: error => Debug.Log(error));
}

private void CreateCube(Entity entity)
{
    if (entity.Flag == EntityFlag.EntityFlagParticipantEntity) return;

    var pose = _conjureKit.GetSession().GetEntityPose(entity);
    var touchableCube = Instantiate(cube, pose.position, pose.rotation).GetComponent<TouchableByHand>();
    _cubes[entity.Id] = touchableCube.GetComponent<Renderer>();
    _cubes[entity.Id].material.color = _colorSystem.GetColor(entity.Id);

    touchableCube.OnTouched += () =>
    {
        _colorSystem.SetColor(entity.Id, Random.ColorHSV());
        _cubes[entity.Id].material.color = _colorSystem.GetColor(entity.Id);
    };
}

Build and run on two or more devices, and you should see cubes with shared colors.

Complete code​

ConjureKitManager.cs:

using System.Collections.Generic;
using UnityEngine;
using Auki.ConjureKit;
using UnityEngine.UI;
using Auki.ConjureKit.Manna;
using Auki.Ur;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using State = Auki.ConjureKit.State;

public class ConjureKitManager : MonoBehaviour
{
    [SerializeField] private Camera arCamera;
    [SerializeField] private ARSession arSession;
    [SerializeField] private ARRaycastManager arRaycastManager;

    [SerializeField] private Text sessionState;
    [SerializeField] private Text sessionID;

    [SerializeField] private GameObject cube;
    [SerializeField] private Button spawnButton;

    [SerializeField] Button qrCodeButton;
    private bool _qrCodeBool;

    private IConjureKit _conjureKit;
    private Manna _manna;

    private ARCameraManager _arCameraManager;
    private Texture2D _videoTexture;

    [SerializeField] private GameObject fingertipLandmark;
    private HandTracker _handTracker;
    private bool _landmarksVisualizeBool = false;

    [SerializeField] private AROcclusionManager arOcclusionManager;
    private bool _occlusionBool = true;

    [SerializeField] private Transform arSessionOrigin;

    private ColorSystem _colorSystem;
    private Dictionary<uint, Renderer> _cubes = new Dictionary<uint, Renderer>();

    void Start()
    {
        _arCameraManager = arCamera.GetComponent<ARCameraManager>();

        _conjureKit = new ConjureKit(
            arCamera.transform,
            "YOUR_APP_KEY",
            "YOUR_APP_SECRET");

        _manna = new Manna(_conjureKit);
        _manna.GetOrCreateFrameFeederComponent().AttachMannaInstance(_manna);

        _conjureKit.OnStateChanged += state =>
        {
            if (state == State.JoinedSession)
            {
                Debug.Log("State.JoinedSession  " + Time.realtimeSinceStartup);
            }

            if (state == State.Calibrated)
            {
                Debug.Log("State.Calibrated  " + Time.realtimeSinceStartup);
            }

            sessionState.text = state.ToString();
            ToggleControlsState(state == State.Calibrated);
        };

        _conjureKit.OnJoined += session =>
        {
            Debug.Log("OnJoined " + Time.realtimeSinceStartup);
            sessionID.text = session.Id.ToString();

            _colorSystem = new ColorSystem(session);
            session.RegisterSystem(_colorSystem, () => Debug.Log("System registered in session"));
            _colorSystem.OnColorComponentUpdated += OnColorComponentUpdated;
        };

        _conjureKit.OnLeft += session =>
        {
            sessionID.text = "";
        };

        _conjureKit.OnEntityAdded += CreateCube;
        _conjureKit.Connect();

        _handTracker = HandTracker.GetInstance();
        _handTracker.SetARSystem(arSession, arCamera, arRaycastManager);

        _handTracker.OnUpdate += (landmarks, translations, isRightHand, score) =>
        {
            if (score[0] > 0 && _landmarksVisualizeBool)
            {
                var handPosition = new Vector3(
                    translations[0],
                    translations[1],
                    translations[2]);

                var pointerLandmarkIndex = 8 * 3; // Index fingertip
                var pointerLandMarkPosition = new Vector3(
                    landmarks[pointerLandmarkIndex + 0],
                    landmarks[pointerLandmarkIndex + 1],
                    landmarks[pointerLandmarkIndex + 2]);

                fingertipLandmark.SetActive(true);

                fingertipLandmark.transform.position =
                    arCamera.transform.TransformPoint(handPosition + pointerLandMarkPosition);
            }
            else
            {
                fingertipLandmark.SetActive(false);
            }
        };

        _handTracker.Start();
    }

    private void Update()
    {
        _handTracker.Update();
    }

    private void ToggleControlsState(bool interactable)
    {
        if (spawnButton) spawnButton.interactable = interactable;
        if (qrCodeButton) qrCodeButton.interactable = interactable;
    }

    public void ToggleLighthouse()
    {
        _qrCodeBool = !_qrCodeBool;
        _manna.SetLighthouseVisible(_qrCodeBool);
    }

    public void ToggleHandLandmarks()
    {
        _landmarksVisualizeBool = !_landmarksVisualizeBool;

        if (_landmarksVisualizeBool)
        {
            _handTracker.ShowHandMesh();
        }
        else
        {
            _handTracker.HideHandMesh();
        }
    }

    public void ToggleOcclusion()
    {
        _occlusionBool = !_occlusionBool;

        arOcclusionManager.requestedHumanDepthMode = _occlusionBool ? HumanSegmentationDepthMode.Fastest : HumanSegmentationDepthMode.Disabled;
        arOcclusionManager.requestedHumanStencilMode = _occlusionBool ? HumanSegmentationStencilMode.Fastest : HumanSegmentationStencilMode.Disabled;
        arOcclusionManager.requestedEnvironmentDepthMode = _occlusionBool ? EnvironmentDepthMode.Fastest : EnvironmentDepthMode.Disabled;
    }

    public void CreateCubeEntity()
    {
        if (_conjureKit.GetState() != State.Calibrated)
            return;

        Vector3 position = arCamera.transform.position + arCamera.transform.forward * 0.5f;
        Quaternion rotation = Quaternion.Euler(0, arCamera.transform.eulerAngles.y, 0);

        Pose entityPos = new Pose(position, rotation);

        _conjureKit.GetSession().AddEntity(
            entityPos,
            onComplete: entity =>
            {
                // Initialize with white color
                _colorSystem.SetColor(entity.Id, Color.white);

                CreateCube(entity);
            },
            onError: error => Debug.Log(error));
    }

    private void CreateCube(Entity entity)
    {
        if (entity.Flag == EntityFlag.EntityFlagParticipantEntity) return;

        var pose = _conjureKit.GetSession().GetEntityPose(entity);
        var touchableCube = Instantiate(cube, pose.position, pose.rotation).GetComponent<TouchableByHand>();
        _cubes[entity.Id] = touchableCube.GetComponent<Renderer>();
        _cubes[entity.Id].material.color = _colorSystem.GetColor(entity.Id);

        touchableCube.OnTouched += () =>
        {
            _colorSystem.SetColor(entity.Id, Random.ColorHSV());
            _cubes[entity.Id].material.color = _colorSystem.GetColor(entity.Id);
        };
    }

    private void OnColorComponentUpdated(uint entityId, Color color)
    {
        _cubes[entityId].material.color = color;
    }
}

ColorSystem.cs:

using System;
using System.Collections.Generic;
using Auki.ConjureKit;
using Auki.ConjureKit.ECS;
using UnityEngine;

/// <summary>
/// The ColorSystem adds and deletes the Color component,
/// maintains and updates a local map with component data 
/// </summary>
public class ColorSystem : SystemBase
{
    // The unique name of the component
    private const string COLOR_COMPONENT_NAME = "color";

    /// <summary>
    /// Triggered when a component data is updated by another participant
    /// </summary>
    public event Action<uint, Color> OnColorComponentUpdated;

    // Local Color component data map
    private readonly IDictionary<uint, Color> _entityColorDataMap = new Dictionary<uint, Color>();

    public ColorSystem(Session session) : base(session)
    {
    }

    // The system will be notified when any component in the returned array is updated or removed
    public override string[] GetComponentTypeNames()
    {
        return new[] { COLOR_COMPONENT_NAME };
    }

    /// Broadcast from the server when another participant updates a Color component with new data.
    public override void Update(IReadOnlyList<(EntityComponent component, bool localChange)> updated)
    {
        foreach (var (entityComponent, localChange) in updated)
        {
            // Update the local data and notify about the update
            _entityColorDataMap[entityComponent.EntityId] = ByteArrayToColor(entityComponent.Data);
            OnColorComponentUpdated?.Invoke(entityComponent.EntityId, _entityColorDataMap[entityComponent.EntityId]);
        }

    }


    /// Broadcast from server when another participant removes a Color component from an entity
    public override void Delete(IReadOnlyList<(EntityComponent component, bool localChange)> deleted)
    {
        foreach (var (entityComponent, localChange) in deleted)
        {
            var entity = _session.GetEntity(entityComponent.EntityId);
            if (entity == null) continue;

            _entityColorDataMap.Remove(entity.Id);
        }
    }



    /// <summary>
    /// Tries to update the Color component data locally and broadcast the update to other participants.
    /// </summary>
    /// <returns> False if entity does not exists, true if component was added/updated successfully.</returns>
    public bool SetColor(uint entityId, Color color)
    {
        // Check if the entity with the given id exists
        var entity = _session.GetEntity(entityId);
        if (entity == null) return false;

        // Store the data locally  
        _entityColorDataMap[entityId] = color;

        // If the entity doesn't already have Color component add one
        var component = _session.GetEntityComponent(entityId, COLOR_COMPONENT_NAME);
        if (component == null)
        {
            _session.AddComponent(
                COLOR_COMPONENT_NAME,
                entityId,
                ColorToByteArray(color),
                () => { },
                error => Debug.LogError(error)
            );

            return true;
        }
        else
        {
            return _session.UpdateComponent(
                COLOR_COMPONENT_NAME,
                entityId,
                ColorToByteArray(color)
            );
        }
    }

    /// <summary>
    /// Get the local Color component data
    /// </summary>
    public Color GetColor(uint entityId)
    {
        if (_session.GetEntity(entityId) == null || !_entityColorDataMap.ContainsKey(entityId))
            return Color.clear;

        return _entityColorDataMap[entityId];
    }

    /// <summary>
    /// Delete the component locally and notify the other participants
    /// </summary>
    public void DeleteColor(uint entityId)
    {
        _session.DeleteComponent(COLOR_COMPONENT_NAME, entityId, () =>
        {
            _entityColorDataMap.Remove(entityId);
        });
    }

    // Convert Color32 to byte array
    private byte[] ColorToByteArray(Color32 color)
    {
        byte[] colorBytes = new byte[4];
        colorBytes[0] = color.r;
        colorBytes[1] = color.g;
        colorBytes[2] = color.b;
        colorBytes[3] = color.a;

        return colorBytes;
    }

    // Convert byte array to Color32 
    private Color32 ByteArrayToColor(byte[] bytes)
    {
        if (bytes.Length < 4)
        {
            Debug.LogError("Byte array must have at least 4 elements (R, G, B, A).");
            return Color.clear;
        }

        byte r = bytes[0];
        byte g = bytes[1];
        byte b = bytes[2];
        byte a = bytes[3];

        Color32 color = new Color32(r, g, b, a);
        return color;
    }
}

TouchableByHand.cs:

using System;
using UnityEngine;

public class TouchableByHand : MonoBehaviour
{
    public event Action OnTouched;

    private void OnTriggerEnter(Collider other)
    {
        if (other.CompareTag("hand"))
        {
            OnTouched?.Invoke();
        }
    }
}

The full project can be found on GitHub.