Serialize large JSON and stream over HTTP in .NET

Question

I am trying to serialize 900000 records with 12 columns each to a JSON object and stream it over HTTP in .NET. I am getting an out of memory exception, as it seems like the object is initially created and then streamed (I realised this by debugging the application). This is my code so far:

public class JsonStreamingResult : ActionResult
{
    private IEnumerable itemsToSerialize;

    public JsonStreamingResult(IEnumerable itemsToSerialize)
    {
        this.itemsToSerialize = itemsToSerialize;
    }

    public override void ExecuteResult(ControllerContext context)
    {
        var response = context.HttpContext.Response;
        response.ContentType = "application/json";
        response.ContentEncoding = System.Text.Encoding.UTF8;
        JsonSerializer serializer = new JsonSerializer();

        object itm;

        using (StreamWriter sw = new StreamWriter(response.OutputStream))
            using (JsonTextWriter writer = new JsonTextWriter(sw))
            {
                writer.WriteStartArray();
                foreach (object item in itemsToSerialize)
                {
                    Newtonsoft.Json.Linq.JObject obj = Newtonsoft.Json.Linq.JObject.FromObject(item, serializer);
                    obj.WriteTo(writer);
                    writer.Flush();
                }

                writer.WriteEndArray();
            }
    }
}

public class HomeController : Controller
{

    public ActionResult Index()
    {
        return View();
    }

    public ActionResult streamMason(int id)
    {
        var masonItems = new List<object>();
        DataContext dc = new DataContext();
        var viewData = dc.Articles.GetPublishedArticles(null, id);

        int cnt = 0;

        for (int i = 0; i < 900000; i++)
        {
            masonItems.Add(new
                    {
                    Field1 = "test1",
                    Field2 = "test2",
                    Field3 = "test3",
                    Field4 = "test4",
                    Field5 = "test5",
                    Field6 = "test6",
                    Field7 = "test7",
                    Field8 = "test8",
                    Field9 = "test9",
                    Field10 = "test10",
                    Field11 = "test11",
                    Field12 = "test12"
                    });
        }

        return new JsonStreamingResult(masonItems);
    }
}

My question: Is it possible to stream it while the object is being generated?

My code is based on this: Streaming large list of data as JSON format using Json.net

EDIT: When I try to serialise with less records (up to ~300000) it works as expected.

By debugging the application I realised that it gets the out of memory after `writer.WriteEndArray();`. It seems like first the object is serialised and afterwards sent over HTTP. Is it possible to do these things in parallel using Async? — Iakovos, May 29 '17 at 13:30

score 2 · Accepted Answer · answered May 29 '17 at 13:43

2

Found the solution and posting for anyone interested:

Just add: response.Flush() after writer.Flush().

answered May 29 '17 at 13:43

Iakovos

1,842
4
25
30

Serialize large JSON and stream over HTTP in .NET

1 Answers1