Big JSON stream on Azure

Question

I'm working on a big data export API, but I'm having isssues when it needs to transport big data as JSON. An example of such is a transfer of over 4 milion records. When saved as a textfile, the data is suposed to be about 380MB, but for some reason the stream is cut short to about 250-280MB (always dfferent) and when I check the file in notepad, it did just cut off the data in the middle of a record.

This behaviour is only happening on the Azure server, I can download the full file through my local IIS. Also weird is that when I export the data as XML, which results in an even bigger file of +600MB did not have this issue.

Our Azure app service plan is S3 (4 cores, 7GB memory) which I believe should be enough, the code that actually transfers the data is the following function:

public IActionResult ResponseConvert(IList data)
{
    return new Microsoft.AspNetCore.Mvc.JsonResult(data);
}

The data parameter is a List<dynamic> object, containing the +4 milion records.

At first glance it seems like Azure terminates the stream prematurely, any idea why and how this can be prevented?

Answer 1

In the end I've writen my own JsonResult class, that would use a JsonTextWriter to transfer the data. This seems to work fine with larger objects, even on Azure.

Here's the full class:

using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;
using System.Collections;
using System.Collections.Generic;
using System.Dynamic;
using System.IO;
using System.Linq;
using System.Text;

namespace MyProject.OutputFormat
{
    public class JsonResult : ActionResult
    {
        private readonly IList _data;

        public Formatting Formatting { get; set; }
        public string MimeType { get; set; }

        public JsonResult(IList data)
        {
            _data = data;


            // Default values
            MimeType = "application/json";
            Formatting = Formatting.None;
        }

        public override void ExecuteResult(ActionContext context)
        {
            context.HttpContext.Response.ContentType = MimeType;
            using (var sw = new StreamWriter(context.HttpContext.Response.Body, Encoding.UTF8))
            {
                using (var writer = new JsonTextWriter(sw) { Formatting = Formatting })
                {
                    writer.WriteStartArray();
                    if (_data != null)
                    {
                        foreach (var item in _data)
                        {
                            writer.WriteStartObject();
                            if (item is ExpandoObject)
                            {
                                foreach (KeyValuePair<string, object> prop in item as ExpandoObject)
                                {
                                    writer.WritePropertyName(prop.Key);
                                    writer.WriteValue(prop.Value != null ? prop.Value.GetType().Name != "Byte[]" ? prop.Value.ToString() : ((byte[])prop.Value).BinaryToString() : null);
                                }
                            }
                            else
                            {
                                var props = item.GetType().GetProperties().Where(i => i.Name != "Item");
                                foreach (var prop in props)
                                {
                                    var val = prop.GetValue(item);
                                    writer.WritePropertyName(prop.Name);
                                    writer.WriteValue(val != null ? val.GetType().Name != "Byte[]" ? val.ToString() : ((byte[])val).BinaryToString() : null);
                                }
                            }
                            writer.WriteEndObject();
                        }
                    }
                    writer.WriteEndArray();
                }
            }
        }
    }
}

The BinaryToString() method you see is a self written extension on byte[] to convert a byte array to a base64 string.

Small note, though this works for bigger data, the JsonResult in Microsoft.AspNetCore.Mvc downloads faster. Getting the response to the client is as fast, but since this method only converts during download, it takes a bit longer until the stream is fully downloaded. If you do not have any issues in your environment, I'd advise using the one in Microsoft.AspNetCore.Mvc.