0

I am building a restful service to serve requests to a client I do not directly control (Couchbase Lite). The client has a bug that makes a request for 7-11k documents at once. It also requires the response to be multipart which is designed as a nested object response type in WebApi.

Using normal datareaders to get the JSON data from a SQL Server as a big batch and passing them to WebApi creates a large number of objects on the large object heap that the .NET app does not free until the site is restarted. Additionally, I have a need to actually insert some data into each object. Parsing the JSON using JSON.net, inserting a property and re-serializing is very slow and actually doubles the number of large objects on the .NET heap.

Is there an efficient way to stream these results without creating tons of large allocations?

im_nullable
  • 555
  • 1
  • 6
  • 17
  • 1
    You can stream directly from an `IDataReader` to a stream using a Json.NET `JsonConverter` as is shown in [JSON.net serialize directly from oledbconnection](https://stackoverflow.com/questions/33835729) and [Dealing with large JSON data returned by Web API](https://stackoverflow.com/questions/38794799). But there may be other levels in the WebAPI stack that buffer the entire response. You'll need to track all of them down and disable them, and I've never done it. Possibly [chunking](http://stackoverflow.com/questions/31388424) might help. – dbc Nov 08 '16 at 00:53
  • 1
    I was hoping for at least one example from someone who has done this end-to-end. Amazing that it's so hard to come across a full example considering how much performance gain there is to be had with these techniques. – im_nullable Nov 08 '16 at 01:55
  • 1
    `StreamContent` might also help, see [WebAPI StreamContent vs PushStreamContent](https://stackoverflow.com/questions/16168683/webapi-streamcontent-vs-pushstreamcontent) and http://blog.guvweb.co.uk/2014/07/02/streaming-web-api/ – dbc Nov 08 '16 at 02:01
  • I though about using `PushStreamContent` but you have to wait for a callback which makes it really hard to read rows from a `IDataReader` and write to a new `PushStream` when it becomes available. The issue is that `MultipartContent` in WebApi expects a collection of *Content objects so each row from the `IDataReader` becomes a new PushStreamContent with an async callback... it gets complex fast. – im_nullable Nov 08 '16 at 02:15
  • What about just [`StreamContent`](https://msdn.microsoft.com/en-us/library/system.net.http.streamcontent(v=vs.118).aspx) then? – dbc Nov 08 '16 at 02:40
  • I thought about that but I don't think it's possible to hang onto streams from an `IDataReader` after you read the next record so creating a collection of `StreamContents` is no good if you cannot guarantee that the stream is around by the time WebApi gets around to reading them which is after the entire collection is created in the case of [MultipartContent](https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/MultipartContent.cs) – im_nullable Nov 08 '16 at 03:19
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/127594/discussion-between-im-nullable-and-dbc). – im_nullable Nov 08 '16 at 06:30

0 Answers0