3

A data intensive intranet application uses the following angularjs post function. It calls a Web API service returning a large JSON object in a chunked stream. This works perfectly well for responses of 100Mb but fails when the response reaches about 250 Mb. 

$scope.refresh = function (){
        usSpinnerService.spin("main-grid");
        $http.post('/api/Assets/GetLargeJSONResult', $scope.postPayload
            ).then(
                function successCallback(response) {
                    $scope.mainGridOptions.api.setRowData(response.data);
                    $scope.mainGridOptions.api.refreshView();
                    clearErrorMessage();
            },
            httpErrorCallback)
            .then(refreshGridTwo)
            .finally(function () {
                usSpinnerService.stop("main-grid");
            });
};

Putting a breakpoint on the line $scope.mainGridOptions.api.setRowData(response.data); shows that response.data is an empty string.

Everything else works fine. I've tried various different ways of returning the JSON response from the web server and so I doubt the problem is on the server end. I'm targeting the Chrome browser in Windows.

Relevant request headers:

  • Accept: application/json, text/plain, /
  • Accept-Encoding: gzip, deflate, br
  • Accept-Language: en-US,en;q=0.8
  • Connection: keep-alive

Relevant response headers

  • HTTP/1.1 200 OK
  • Cache-Control: no-cache
  • Transfer-Encoding: chunked
  • Content-Type: application/json
  • Expires: -1

I know one option would be to page the response into smaller chunks. However, I would prefer to keep the data intact as it's being analysed in a grid. I wondered if I'm missing something as I can successfully stream very large open office xml files (media type "application/octet-stream") to the browser from Web API without any issues.

Community
  • 1
  • 1
steve_cdi
  • 166
  • 6
  • 1
    Similar questions: http://stackoverflow.com/q/28710394/215552, http://stackoverflow.com/q/22103927/215552, http://stackoverflow.com/q/23181492/215552 – Heretic Monkey Mar 13 '17 at 22:28
  • Have you put a breakpoint where it actually returns the value in web api to make sure its serializing correctly? I've hit a point where it returns but hits a serialization error. – Dylan Mar 14 '17 at 01:51
  • Dylan, There are no serialization errors. It returns correctly for objects of the same type but around 100Mb. It's due to the size. The similar questions posted by Mike (thanks!) all point to consuming the stream one chunk at a time, but there doesn't seem to be a simple 'angular' way of doing it. – steve_cdi Mar 14 '17 at 13:34
  • Possibly a limit set by the web server. Have you tried accessing the data using an external tool such as `curl`, `wget`, etc.? – mhawke Mar 14 '17 at 21:55
  • Check the value set for `maxAllowedContentLength` in your web.config. You can also check this link: http://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/ – mhawke Mar 14 '17 at 22:51
  • Update on 'progress' so far: I used oboe.js to parse the streamed result with some success in that I can at least see that the data arriving one batch at a time. If I push the results into an array, I still get an 'aw snap' out of memory error in Chrome even though I use oboe's oboe.drop; method. Perhaps 200Mb is just too much data in one array for Chrome? At least this rules out the server as the problem. – steve_cdi Mar 14 '17 at 23:41
  • This seems to be relevant... http://stackoverflow.com/questions/36043725/loading-large-json-files-250mb-with-xmlhttprequest – steve_cdi Mar 14 '17 at 23:50
  • Also this http://stackoverflow.com/questions/37402716/handle-xmlhttprequest-response-with-large-data – steve_cdi Mar 15 '17 at 04:19
  • In Chrome 85 XMLHttpRequest#response returns "null" and no error event is fired when loading >512MB json. Fetch API (with Response#json()) throws an error with confusing message "SyntaxError: Unexpected end of JSON input". This looks like a browser bug. – Viktor Mukhachev Oct 01 '20 at 14:56

0 Answers0