How to deserialize multiple objects from a json stream?

Question

Ok, I have asked questions about something like this before, but this is a different subject, so i feel i should make a new topic about it. I'm sorry if this is something i should not have done...

Anyway:

I'm currently reading a twitterfeed and trying to convert it to lose (status) objects. The code i have now is as follows but fails:

webRequest = (HttpWebRequest)WebRequest.Create(stream_url);
webRequest.Credentials = new NetworkCredential(username, password);
webRequest.Timeout = -1;
webResponse = (HttpWebResponse)webRequest.GetResponse();
Encoding encode = Encoding.GetEncoding("utf-8");
responseStream = new StreamReader(webResponse.GetResponseStream(), encode);
int i = 0;

//Read the stream.
while (_running)
{
    jsonText = responseStream.ReadLine();

    byte[] sd = Encoding.Default.GetBytes(jsonText);
    stream.Write(sd, i, i + sd.Length);

    try
    {
        status s = json.ReadObject(stream) as status;
        if (s != null)
        {
            //write s to a file/collection or w/e
            i = 0;
        }
    }
    catch
    {

    }

}

The idea is: Copy the stream into another stream. and keep trying to read it untill an status object is discovered. This was ment to prevent the stream for being to little, so it had chance to grow. Ofcourse the stream does not always start at the start of an object, or can be corrupt.

Now i did find the method IsStartObject, and i think i should use it. Though i have no experience with streams and i can never find a good example of how to use this.

Is there anyone who can explain to me how to read multiple objects from the stream so i can write them into a list or w/e. I really can't find any good examples on the internets..

Thank you very much for trying!!!

I am not sure that I can understand, why dont you read all response `responseStream.ReadToEnd()` and deserialize to a class (or parse)? — L.B, Jan 16 '12 at 22:06
It's an endless stream (the twitter feed). And the Status object is a class i want to deserialize to. But the stream keeps streaming these objects (without end). — Jasper, Jan 16 '12 at 22:07
Sure! just use https://stream.twitter.com/1/statuses/sample.json in combination with any twitter user/pass — Jasper, Jan 16 '12 at 22:35

score 1 · Accepted Answer · answered Jan 16 '12 at 23:42

1

I used Json.Net library and this extension class that makes use of DynamicObject to parse streaming json objects

HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://stream.twitter.com/1/statuses/sample.json");
webRequest.Credentials = new NetworkCredential("...", "......");
webRequest.Timeout = -1;
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
Encoding encode = Encoding.GetEncoding("utf-8");
StreamReader responseStream = new StreamReader(webResponse.GetResponseStream());

string line;
while (true)
{
    line = responseStream.ReadLine();
    dynamic obj = JsonUtils.JsonObject.GetDynamicJsonObject(line);
    if(obj.user!=null)
        Console.WriteLine(obj.user.screen_name + " => " + obj.text);
}

answered Jan 16 '12 at 23:42

L.B

114,136
19
178
224

Very usefull answer! Thank you very much, this is exactly what i needed:) One question though: can it happen that the parsing to a dynamic json object happends slower then the stream is comming in, which wil result in the stream being buffered in memory? If so: how can i solve this? – Jasper Jan 17 '12 at 10:01
It is a producer-comsumer problem. You can run the reader in a separate thread, put the `line`s to a queue(ex, BlockingQueue) and the main thread can read the lines from that queue and create a dynamic object(you can even process the `line`s in BlockingQueue with many threads) – L.B Jan 17 '12 at 10:08
Thanks for your help. One more noob question though: is it normal that one object is in one line? When i started i was just expecting parts of data...like it reads untill half way an objects, then next line is next part etc...is it always this easy or is that just different depending on the provider of the stream? – Jasper Jan 17 '12 at 21:32
It depends on the provider. Twiitter makes the life easy for you. – L.B Jan 17 '12 at 21:51
Makes me wonder how to do it if it was the hell i discribed....are there examples/tutorials that you know of? – Jasper Jan 17 '12 at 22:24
1

I don't think. It is application/protocol/developer dependent. But if you talk of streaming json objects(as a pure text), you can try to count the nesting levels of `{` and `}`. Start with 0, increment the counter with `{` and decrement with `}`. if the count is zero you have reached the end of the object and you can deserialize it (Just a quick thought) – L.B Jan 17 '12 at 22:30
any idea how to do this in .net 3.5? – KamalSalem Mar 08 '12 at 19:54

score -1 · Answer 2 · edited May 23 '17 at 12:19

This is an implementation of L.B's suggestion for splitting by counting the nesting level of { and }.

public static IEnumerable<string> JsonSplit(this StreamReader input, char openChar = '{', char closeChar = '}',
  char quote='"', char escape='\\')
{
  var accumulator = new StringBuilder();
  int count = 0;
  bool gotRecord = false;
  bool inString = false;
  while (!input.EndOfStream)
  {
    char c = (char)input.Read();
    if (c == escape)
    {
      accumulator.Append(c);
      c = (char)input.Read();
    }
    else if (c == quote)
    {
      inString = !inString;
    }
    else if (inString)
    {
    }
    else if (c == openChar)
    {
      gotRecord = true;
      count++;
    }
    else if (c == closeChar)
    {
      count--;
    }
    accumulator.Append(c);
    if (count != 0 || !gotRecord) continue;
    // now we are not within a block so 
    string result = accumulator.ToString();
    accumulator.Clear();
    gotRecord = false;
    yield return result;
  }
}

Here's a test

[TestClass]
  public class UnitTest1
  {
    [TestMethod]
    public void TestMethod1()
    {
      string text = "{\"a\":1}{\"b\":\"hello\"}{\"c\":\"oh}no!\"}{\"d\":\"and\\\"also!\"}";
      var reader = FromStackOverflow.GenerateStreamFromString(text);
      var e = MyJsonExtensions.JsonSplit(reader).GetEnumerator();
      e.MoveNext();
      Assert.AreEqual("{\"a\":1}", e.Current);
      e.MoveNext();
      Assert.AreEqual("{\"b\":\"hello\"}", e.Current);
      e.MoveNext();
      Assert.AreEqual("{\"c\":\"oh}no!\"}", e.Current);
      e.MoveNext();
      Assert.AreEqual("{\"d\":\"and\\\"also!\"}", e.Current);
    }
  }

The implementation of GenerateStreamFromString is here

This would fail if your openChar/closeChar happens to be in a string. E.G. {"a":1}{"b":"hello"}{"c":"oh}no!"} — Graeme Job, Jun 14 '16 at 12:15
Thanks @GraemeJob. Whilst not wishing to write a json parser I've updated my answer to handle the obvious gotchas. — jdpilgrim, Oct 06 '16 at 10:56

How to deserialize multiple objects from a json stream?

2 Answers2

Linked