Returning an extremely large JSON (1GB)

Question

I have RESTful API that returns some large JSON files. In some instance, the returned string could become more than 1GB. When that happens, I only get some part of the content, meaning, I'm returning an array of objects, but only the first item is returned. Is there a way to remove this limitation? Here's my code (which is pretty simple).

[HttpPost]
[Route("rest/result")]
public IActionResult GetResult(string requestId)
{
    var item = _service.GetItem(requestId);
    return item?.Result == null ? NotFound() : Ok(item.Result);
}

Edit:

This seems to return the 1GB JSON string but I'm not sure it's exactly the same data:

[HttpPost]
[Route("rest/result")]
public IActionResult GetResult(string requestId)
{
    var item = _service.GetItem(requestId);
    if (item?.Result == null)
    {
        return NotFound();
    }

    var json = JsonSerializer.Serialize(item.Result);
    return Content(json, "application/json");
}

How are you checking the response size to know only the first item is returned? — Matt Thomas, Sep 21 '22 at 19:53
As an aside... In this code is `.Result` a property on a `Task<>`? Or just coinsidence that the type returned here has a similarly named property? — David, Sep 21 '22 at 19:54
1GB of text is a truly massive amount of text. Are you certain that you can't simply get less data in the first place? — Joe, Sep 21 '22 at 20:28
@MattThomas I save the file on the server before sending it to the client. The response size is more or less 1GB for this particular request. — Alireza Noori, Sep 21 '22 at 20:31
I set the mentioned option on the other post but it didn't change anything. — Alireza Noori, Sep 21 '22 at 20:32
@Joe I wrote this API but I can't have much of a say in the data that they produce on the other side. I suggested that to the other sections of the company but I just want to fix it if possible. — Alireza Noori, Sep 21 '22 at 20:33
@David similar name but no, it's a property named "Result" not a result of a Task. — Alireza Noori, Sep 21 '22 at 20:34
I still wonder if you're running into some limitation with the JSON serializer. [Example](https://learn.microsoft.com/en-us/dotnet/api/system.text.json.jsonserializeroptions.maxdepth?view=net-6.0#system-text-json-jsonserializeroptions-maxdepth). Perhaps try a different serializer to see what effect that has? — Matt Thomas, Sep 21 '22 at 20:39
You are going to run into all manner of issues with such large blob of text. Firewalls might choke on it, and web servers don't handle it well either, and clients may get out of memory exceptions if they are not streaming the content properly. Avoid it if you can — Charlieface, Sep 22 '22 at 00:26
@MattThomas I'm not sure which method the `Ok(obj)` is using to create the JSON string but `JsonSerializer.Serialize(item.Result)` is returning a 1GB string without any issues. Do you know if it's different or not? I assumed that it should be something similar. — Alireza Noori, Sep 22 '22 at 00:59
@Charlieface I totally agree with you and my colleagues said that they're going to break up the data into chunks but I'd like to know what's enforcing that limitation and how I can fix that. — Alireza Noori, Sep 22 '22 at 01:00
Like I said, it could be anything along the way. Usual suspects are firewall or any other proxy, web server settings, web server or client memory capacity (this last is unlikely as you would have got an exception). — Charlieface, Sep 22 '22 at 09:17

score 0 · Accepted Answer · answered Feb 18 '23 at 13:51

I think you are, hitting a default response size limit. so to adjust the limit you could use ResponseBufferSize property in your startup.configureServices

services.Configure<IISServerOptions>(options =>
{
    options.MaxRequestBodySize = int.MaxValue;
    options.MaxRequestBufferSize = int.MaxValue;
    options.MaxResponseBufferSize = int.MaxValue;
});

but, returning extremely large file could reduce the performance and scalability of your application. the other ways to handle your result is to use pagination by dividing your result in small chunks or zip your result by compressing it.

Returning an extremely large JSON (1GB)

Edit:

1 Answers1