How to create a MutableRuntimeReferenceImageLibrary in Unity?

Question

I'm using Unity 2021.3.22f1 and the version 4.2.7 of AR Foundation. I've already added to my scene the AR Session and AR Session Origin, and also AR Tracked Image Manager as a component of AR Session Origin.

I would like to create a MutableRuntimeReferenceImageLibrary which, runtime, add itself dynamically images that are contained in a project's folder.

I wrote this snippet of code:

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

public class ImageLibraryManager : MonoBehaviour
{
    public ARTrackedImageManager trackedImageManager;
    public string imageFolderPath;

    void Start()
    {
        // Create a list to hold XRReferenceImage objects
        var imageList = new List<XRReferenceImage>();

        // Get the files in the image folder
        var imageFiles = System.IO.Directory.GetFiles(imageFolderPath);

        // Loop through the image files and add them to the list
        foreach (var imageFile in imageFiles)
        {
            // Load the image from the file
            var imageBytes = System.IO.File.ReadAllBytes(imageFile);
            var texture = new Texture2D(2, 2);
            texture.LoadImage(imageBytes);

            var newGuid = System.Guid.NewGuid();
            var guidBytes = newGuid.ToByteArray();
            var image = new XRReferenceImage(
                // Use a new guid for each image
                new SerializableGuid(
                    (ulong)(guidBytes[3] << 24 | guidBytes[2] << 16 | guidBytes[1] << 8 | guidBytes[0]),
                    (ulong)(guidBytes[7] << 24 | guidBytes[6] << 16 | guidBytes[5] << 8 | guidBytes[4])
                ),
                // Use an empty guid for the second parameter
                SerializableGuid.empty,
                // Set the size of the image in meters
                new Vector2(texture.width, texture.height) / 1000f,
                // Set the name of the image
                System.IO.Path.GetFileNameWithoutExtension(imageFile),
                // Set the texture of the image
                texture
            );

            imageList.Add(image);
        }

        // Create a new MutableRuntimeReferenceImageLibrary and assign it to the ARTrackedImageManager
        var imageLibrary = trackedImageManager.CreateRuntimeLibrary() as MutableRuntimeReferenceImageLibrary;
        foreach (var image in imageList)
        {
            // Add each image to the image library
            imageLibrary.AddReferenceImage(image);
        }
        trackedImageManager.referenceLibrary = imageLibrary;
    }
}

However, this gives to me this error: Assets/ImageLibraryManager.cs(53,26): error CS1061: 'MutableRuntimeReferenceImageLibrary' does not contain a definition for 'AddReferenceImage' and no accessible extension method 'AddReferenceImage' accepting a first argument of type 'MutableRuntimeReferenceImageLibrary' could be found (are you missing a using directive or an assembly reference?)

I've already tried with other methods like "Add" or "AddImage", but it always gives to me the same error.

Can you help me to solve my issues and to create this type of MutableRuntimeReferenceImageLibrary?

Thank u

Answer 1

It looks like there are a hierarchy of classes for image libraries in Unity. Notice after further reading that RuntimeReferenceImageLibrary and XRReferenceImageLibrary both implement the IReferenceImageLibrary interface. One is meant for edit-time. The other is meant for run-time. Confusing? Absolutely! This is the toughest question I ever answered!

Object
  RuntimeReferenceImageLibrary - create at run-time (but must be casted from Mutable*)
    MutableRuntimeReferenceLibrary - mutable at runtime (add images)
  XRReferenceImageLibrary - create at edit-time; immutable at run-time (don't add images).

Also notice that Unity uses a term of "runtime". Don't get this confused with the common programming term "runtime" when a program is running from the perspective of executing your program via .NET Common Language Runtime. Think of the Unity term from the perspective of being inside the application since you can either be editing or running. Because they have another term called "edit-time". See the link below. The status of the application in "edit-time" versus "runtime" affects how these image libraries function. I think the developers FAILED to use a hyphen, so they can confuse people out of their mind! They should rename it to "run-time"!

https://docs.unity.cn/Packages/com.unity.xr.arfoundation@4.1/api/UnityEngine.XR.ARFoundation.ARTrackedImageManager.html?q=RuntimeReferenceImageLibrary

See the interface documentation.

An IReferenceImageLibrary can be either an XRReferenceImageLibrary or a RuntimeReferenceImageLibrary. XRReferenceImageLibrarys can only be constructed at edit-time and are immutable at runtime. A RuntimeReferenceImageLibrary is the runtime representation of a XRReferenceImageLibrary and may be mutable at runtime (see MutableRuntimeReferenceImageLibrary).

https://docs.unity.cn/Packages/com.unity.xr.arfoundation@4.1/api/UnityEngine.XR.ARFoundation.ARTrackedImageManager.html?q=RuntimeReferenceImageLibrary#UnityEngine_XR_ARFoundation_ARTrackedImageManager_referenceLibrary

CreateRuntimeLibrary(XRReferenceImageLibrary) Creates a UnityEngine.XR.ARSubsystems.RuntimeReferenceImageLibrary from an existing UnityEngine.XR.ARSubsystems.XRReferenceImageLibrary or an empty library if serializedLibrary is null. Use this to construct reference image libraries at runtime. If the library is of type MutableRuntimeReferenceImageLibrary, it is modifiable at runtime.

Cast your MutableRuntimeReferenceImageLibrary to a RuntimeReferenceImageLibrary. But since MutableRuntimeReferenceLibrary class is abstract, it requires a subclass to instantiate an object. There is already a propery called referenceLibrary on the ARTrackedImageManager class. So use that, rather than calling CreateRuntimeLibrary method in this situation. (That is the trouble with frameworks--things are hidden like this unless you read for four hours) Find the properties link on the right side of the link above. You will see referenceLibrary.

RuntimeReferenceImageLibrary runtimeLibrary = trackedImageManager.referenceLibrary as MutableRuntimeReferenceImageLibrary;

After this, I don't know what else is required.

From what I can tell, class XRReferenceImageLibrary is the only image library that has a List method inside, meaning it contains a collection of images. So you might need to build this first, and then convert it to another image library type/class object. See IReferenceImageLibrary documentation above. So you may need to cast the object twice after an XRReferenceImageLibrary is built (images added to it).

The definition has MANY MANY features! The : is used in C# to denote class inheritance or interface implementation. You can enumerate, you can utilize interface functionality, it has a callback receiver, etc. Do I know what these mean? Not really, except enumerating. That means you can access a list/collection of objects (images in this case).

public class XRReferenceImageLibrary : ScriptableObject, IReferenceImageLibrary, ISerializationCallbackReceiver, IEnumerable<XRReferenceImage>, IEnumerable

https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.2/api/UnityEngine.XR.ARSubsystems.XRReferenceImageLibrary.html#methods

It looks like the developers are hiding information so you pay for training to learn the framework. Trust me, I've been programming for 18 years. Many software companies don't share details of their code unless you pay them. You learn things really fast when you pay for training.

If you truly want the source code to confirm your own code, you could download this. Compile as a class library (DLL). Then add a reference to the DLL in your C# application project. Then when you compile and execute your code, you can read their code to answer your own questions with the stack trace line by line in the IDE.

https://github.com/needle-mirror/com.unity.xr.arfoundation

Good luck! If I get more time, I will look at this more.

Answer 2

EDIT (8/25/23): This is working perfectly. The only reason I was having issues with Image Recognition was the "Non Power of 2" setting on the source texture needs to be set to none, per the documentation.

I am working through this process myself as well. I am currently able to add images to my mutable library, and receive back confirmation that they have been added successfully, but am finding the library does not currently work as intended, so take this as a suggestion, not a complete answer. I am working with an uncommon situation of receiving the image to be added in the format of a byte[], so need to complete additional steps, but you should be able to generalize this.

if (arTrackedImageManager.descriptor.supportsMutableLibrary)
{
       var mutableLibrary = arTrackedImageManager.CreateRuntimeLibrary() as MutableRuntimeReferenceImageLibrary;
       AddImagesToLibrary(mutableLibrary, ReconstructLibrary(serializedTextures));
       InitializeNativeImageManager(mutableLibrary);
}            

---

    private Dictionary<Texture2D, XRInfo> ReconstructLibrary(List<SerializableTexture2D> serializedTextures)
    {
        //unique code to my use case here. returns a dictionary, with each key/value pair consisting of a texture2D/XRInfo with XRInfo looking like this:
        //public struct XRInfo
       //{
           //public string name;
           //public bool specifySize;
           //public Float2 size;
       //}
    }

    private void AddImagesToLibrary(MutableRuntimeReferenceImageLibrary mutableLibrary, Dictionary<Texture2D, XRInfo> reconstructedImages)
    {
        foreach (KeyValuePair<Texture2D, XRInfo> entry in reconstructedImages)
        {
            try
            {

                Texture2D newImageTexture = entry.Key;
                string newImageName = entry.Value.name;
                float? newImageWidthInMeters = entry.Value.specifySize ? entry.Value.size.x : null;

                AddReferenceImageJobState jobState = mutableLibrary.ScheduleAddImageWithValidationJob(
                    newImageTexture, 
                    newImageName, 
                    newImageWidthInMeters
                );

                JobHandle jobHandle = jobState.jobHandle;
                jobHandle.Complete();

                if (jobState.status == AddReferenceImageJobStatus.Success)
                {
                    Debug.Log($"Image {newImageName} added to library successfully.");
                }
                else
                {
                    //should report status "ErrorInvalidImage" if arcore rejects image
                    Debug.LogWarning($"Failed to add image {newImageName} to library. {jobState.status}");
                }
            }
            catch (Exception e)
            {
                Debug.LogError($"Failed to add image {entry.Value.name} to library. {e}");
            }
        }
    }

    private void InitializeNativeImageManager(MutableRuntimeReferenceImageLibrary mutableLibrary)
    {
        arTrackedImageManager.enabled = false;
        arTrackedImageManager.referenceLibrary = mutableLibrary;
        arTrackedImageManager.enabled = true;
    }

Note, the image tracking subsystem will only accept certain texture formats to be added at runtime. The following snippet will let you determine which formats can be added:

int supportedFormatCount = 
mutableLibrary.supportedTextureFormatCount;
Debug.Log($"Mutable library supports {supportedFormatCount} texture formats.");
for (int i = 0; i < supportedFormatCount; i++)
{
     TextureFormat supportedFormat = mutableLibrary.GetSupportedTextureFormatAt(i);
     Debug.Log($"Supported Texture Format {i}: {supportedFormat}");
}

This is using the extension method that accepts a texture2d as an argument in place of a native slice.