Reading large JSON file without hanging in Unity

Question

I am developing a mobile game where I need to load a 4D 60X40X60X40 integer array of about 17MB. I am currently reading the data in JSON and deserialising in Unity using SimpleJSON.cs .

The problem arises when parsing the string using SimpleJSON. The game hangs and takes about 30 seconds to load the file.

I have tried using IJobs but they do not allow usage of strings (non-blittable).

The main priority is the hanging part. It does not allow me to give feedback to the user while they wait for the JSON to load. Of course, lowering the loading time and any tips surrounding it would be really helpful.

My code:

IEnumerator StartLoadFiles()
{
    // Wait to give time for the game to load everything else
    yield return new WaitForSeconds(1f);

    /// First read the transition matrices. We are using TextAsset
    /// because of compatibility issues with Android.
    TextAsset file;
    yield return file = Resources.Load(transitionPath) as TextAsset;

    loadingSlider.value = 0.3f;
    string transitionString = file.ToString();
    loadingSlider.value = 0.5f;

    /// We are using SimpleJSON.cs as of now. Maybe there is
    /// something faster out there... (Here is where it hangs)
    JSONNode N;
    yield return N = JSON.Parse(transitionString);

    loadingSlider.value = 0.7f;
}

Edit: I am already using a coroutine with no difference.

You can try [Array Pooling](https://adamsitnik.com/Array-Pool/). I am not sure if that will work efficiently in 4 dimensional array but, maybe if that give any performance boost in your case, you can seperate your 4D array to 4 piece 1D array. — SeLeCtRa, Feb 26 '21 at 20:13
Also, you can use Json.net, it supports deserializing directly from stream. [Deserializing from stream](https://stackoverflow.com/a/32236563/9969193) — SeLeCtRa, Feb 26 '21 at 20:32

derHugo · Accepted Answer · 2021-02-27T12:49:35.490

Note that a Coroutine itself is not async! It still runs every iteration of the IEnumerator in the Unity main thread right after Update.

You would rather use a Task/Thread like e.g. (assuming the first part of the text asset loading works as expected)

private int[,,,] loadValues = null;


IEnumerator StartLoadFiles()
{
    // Wait to give time for the game to load everything else
    yield return new WaitForSeconds(1f);

    TextAsset file;
    yield return file = Resources.Load(transitionPath) as TextAsset;

    loadingSlider.value = 0.3f;
    string transitionString = file.ToString();
    loadingSlider.value = 0.5f;

    // Somehow you have to wait for callbacks into the main thread
    // simplest way is a thread-safe Queue
    var actions = new ConcurrentQueue<Action>();

    // Now start the thread and wait for results
    var thread = new Thread(() => LoadingThread(actions));
    thread.Start();

    while(loadValues == null)
    {
        // process the actions in the maint hread 
        while(actions.Count > 0 && actions.TryDequeue(out var action)
        {
            action?.Invoke();
        }  

        yield return null;
    } 

    Debug.Log("Finished loading");

    // Now o something with "loadValues"
}

private void LoadingThread(ConcurrentQueue actions)
{
    try
    {
        var N = JSON.Parse(transitionString);

        actions.Enqueue(() => {loadingSlider.value = 0.7f;});

        var output = new int[60,40,60,40];

        for(var x = 0; x < 60; x++)
        {
            for(var y = 0; y < 40; y++)
            {
                for(var z = 0; z < 60; z++)
                {
                    for(var w = 0; w < 40; w++)
                    {
                        // Depending on your json format of course
                        output[x,y,z,w] = N[x][y][z][w];
                    }
                }
            }
        }  

        actions.Enqueue(() => {
            loadValues = output;
            loadingSlider.value = 1f;
        });
    }
    catch(Exception e)
    {
        Debug.LogException(e);
        actions.Enqueue(() => {loadValues = new int[0,0,0,0];});
    }
}

Note: Typed on smartphone but I hope the idea gets clear

You are an actual God and I do not deserve you. Thank you so much! It worked like a charm and even made the loading almost 20% faster. Any tips on how to make the loading even faster? — Vaggelis Stamkopoulos, Feb 28 '21 at 16:59
@VaggelisStamkopoulos No I'm not ^^ Well also the `Resources.Load` and `TextAsset.ToString` are quite expensive if your file is that huge. So if there is any possibility to rather load them completely async in the thread via direct FileIO rather use that instead! Actually [**don't use `Resources`**](https://learn.unity.com/tutorial/assets-resources-and-assetbundles#5c7f8528edbc2a002053b5a7) at all if there is any other way ;) E.g. put your file in the `StreamingAssets` and use a streamed deserialization .. this way you wouldn't even have to load the entire content into memory first — derHugo, Feb 28 '21 at 20:38

Reading large JSON file without hanging in Unity

1 Answers1

Linked