Error deserialising string into JSON object

Question

I have a php file which reads a .txt file and sends it via a php server to a c# unity script. Below is a snippet of the text file showing the first 3 lines:

{ "lemma" : "aljotta", "gloss" : "Fisħ soup" }
{ "lemma" : "arguzin", "gloss" : "Slave driver" }
{ "lemma" : "armunjaka", "gloss" : "Armunjaka" }

This is the php script:

<?php
$file = fopen("lemmas.txt", "r");
echo fread($file, filesize("lemmas.txt"));
fclose($file);
?>

In a c# script, the text is returned and each line is separated into an array (string[] lines) slot as seen below:

IEnumerator GetTextFromFile()
    {
        bool succcessful = true;
        WWWForm form = new WWWForm();
        WWW www = new WWW("http://localhost:9000/tounity.php", form);

        yield return www;

        if(www.error != null)
        {
            succcessful = false;
        }
        else
        {
            succcessful = true;
        }

        if (succcessful)
        {
            populateWordList(www.text);
        }
    }

 void populateWordList(string text)
    {
        string[] textArray = text.Split('\n');
        wordsList = gameDatabase.GetWords(textArray);
    }

The array is then passed to a method which deserializes each line into an object of class GameDatabase as seen in the image below:

public string lemma { get; set; } 
public string gloss { get; set; }

public GameDatabase(string lemma, string gloss)
{
    this.lemma = lemma;
    this.gloss = gloss;
}

public  ArrayList GetWords(string[] lines)
{
    foreach (string line in lines)
    {

        GameDatabase gd = JsonConvert.DeserializeObject<GameDatabase>(line);
        lemmasAndGlossesList.Add(new GameDatabase(gd.lemma, gd.gloss));
    }

    foreach(GameDatabase line in lemmasAndGlossesList)
    {
        Debug.Log(line.lemma + "------" + line.gloss);
    }

    return lemmasAndGlossesList;
}

The error occurs in GameDatabase gd = JsonConvert.DeserializeObject<GameDatabase>(line); and returns

JsonReaderException: Unexpected character encountered while parsing value: . Path '', line 0, position 0.

I have searched extensively, however, haven't found anything that works. Any help would be greatly appreciated. It is worth noting that this problem doesn't happen when loading the text file directly into unity without using php.

EDIT

When using the vs debugger this is the value in the line to be deserialized:

The JSON visualiser of Visual Studio 2019 however reports this:

Your JSON llooks fine (as long you read line by line.The usual line break characte s are: `\r\n`. You are just splitting on `\n` leaving a remaining `\r`. Before you deserialize the `line` call: `line = line.Trim('\r')`. — BionicCode, Sep 14 '19 at 16:06
@bassxzero He#s processing each line separately. Each line is a valid JSON. — BionicCode, Sep 14 '19 at 16:07
I tried line.Trim('\r') however this doesn't work. I also tried to add square brackets to the JSON which didn't work and tested out my json in jsonlint.com which returned valid json. — Andre Mousu, Sep 14 '19 at 16:11
As I said trying out the same file without sending it via php and reading directly using a stream in c# works. Could it be that I need to pass any headers with my php? — Andre Mousu, Sep 14 '19 at 16:13
Is there a newline at The end of the file, or any extra newlines in `www.Text`? You may be getting an empty entry when you split on \n... Try printing, or examining in the debugger, the value of `line` or just look at the length of line compared to the number of line that you have the file — Jonathon K, Sep 14 '19 at 16:38
I have edited my post to include screenshots of the debugger @JonathonK — Andre Mousu, Sep 14 '19 at 16:57
Is there a UTF-8 marker on the file? It doesn't print... What do you get if you print line[0] or if you convert the the string to hex like this answer https://stackoverflow.com/a/16999640/8723329 — Jonathon K, Sep 14 '19 at 17:16
This is the hex value I get : EFBBBF7B20226C656D6D6122203A2022617267757A696E222C2022676C6F737322203A2022536C6176652064726976657222207D Note that i removed the dashes as seen in the example you sent. The file is in utf-8 and i also tried manually specifing the header of the php file as both text/html, text/plain. Unfortunately i need utf 8 as some chars i require do not have utf 8 encoding — Andre Mousu, Sep 14 '19 at 17:22

score 3 · Accepted Answer · answered Sep 14 '19 at 19:28

Thanks to Jonathon K's comment and your reply we can see the data returned by the PHP script starts with a BOM: the first three bytes. This nice article explains how to handle such data properly. In short: use a StreamReader to read the data.

This little program demonstrates how it could work with your data:

using System;
using Newtonsoft.Json;
using System.IO;


public class Program
{
    public static void Main()
    {
        var bytes = new byte[] {
            0xEF,0xBB,0xBF,0x7B,0x20,0x22,0x6C,0x65,0x6D,0x6D,0x61,0x22,
            0x20,0x3A,0x20,0x22,0x61,0x72,0x67,0x75,0x7A,0x69,0x6E,0x22,
            0x2C,0x20,0x22,0x67,0x6C,0x6F,0x73,0x73,0x22,0x20,0x3A,0x20,
            0x22,0x53,0x6C,0x61,0x76,0x65,0x20,0x64,0x72,0x69,0x76,0x65,
            0x72,0x22,0x20,0x7D};

        string json;
        using(var ms = new MemoryStream(bytes))
        using(var sr = new StreamReader(ms))
        {
            json = sr.ReadToEnd();
            Console.WriteLine(json);
        }

        // I'm using dynamic here. In your case you can use GameDatabase
        dynamic obj = JsonConvert.DeserializeObject(json);
        Console.WriteLine(obj.lemma);
    }
}

Output:

{ "lemma" : "arguzin", "gloss" : "Slave driver" }
arguzin

score 0 · Answer 2 · answered Sep 14 '19 at 16:17

I dont know the c# sintax but this will work.

change your JSON file.

[
    { "lemma" : "aljotta", "gloss" : "Fisħ soup" },
    { "lemma" : "arguzin", "gloss" : "Slave driver" },
    { "lemma" : "armunjaka", "gloss" : "Armunjaka" }
]

apply JsonConvert.DeserializeObject to www.text

for (GameDatabase line in JsonConvert.DeserializeObject<GameDatabase[]>(www.text)){
    Debug.Log(line.lemma + "------" + line.gloss);
}

Maybe my c# syntax is wrong but i would that u understand my idea

No use. I get the same problem as before. My json is correct as I am reading line by line rather than reading the whole text file at once. — Andre Mousu, Sep 14 '19 at 16:27

score 0 · Answer 3 · answered Sep 14 '19 at 18:15

I think it's possible that you're going about deserialization wrong by using JsonConvert.

Instead, read up on this documentation and try to use the Unity functions: https://docs.unity3d.com/Manual/JSONSerialization.html

For starters, you're defining lemma and gloss incorrectly if you're looking to use them for deserializing JSON in Unity. See this answer for more info: Serialize and Deserialize Json and Json Array in Unity

Error deserialising string into JSON object

3 Answers3