REST API Put large data

Question

I have a REST API implemented in microsoft Web API.
In my client i use HttpRequestMessage and HttpResponseMessage.
When i am sending a small class, I serialize it to JSON and then send it.
Soon, my class becomes bigger, and I need to JSON the class, zip it (in memory) and send to server. I can no longer use the same technique, I need to send the zip in chunks.

What is the proper way to achieve it ? I have read this post Posting a File and Associated Data to a RESTful WebService preferably as JSON

Need some good articles, I dont know where to start.

What kind of data is it and how big it may become? Do you have control over the REST API so that you can modify the serialization because JSON is not well adapted to large data payloads? — Darin Dimitrov, Aug 26 '13 at 14:26
json works just fine for bid data Darin. The only issue is you need to stream parse it with a parser similar to how you use sax parser on large xml. — Daniel Moses, Aug 26 '13 at 14:32
It depends on what you mean by big. If you are to send huge binary files, JSON is probably the last serialization mechanism I would choose. — Darin Dimitrov, Aug 26 '13 at 14:36
One file i am sending is a zipped DB file (SQL DB), I am going to limit this to 1GB. the other file (one i wrote the question about) is class that contain many classes inside, i guess the max of that one could reach 10MB, zipped in memory. — ilansch, Aug 26 '13 at 14:36
Alright, 1GB is pretty large indeed. What about my other question? Can you modify the server so that it doesn't use JSON but it can accept some other formats? — Darin Dimitrov, Aug 26 '13 at 14:38
Right now i can skip the 1GB file. sorry for harassment, My only question is regarding to take a large class (can be 10 MB), serialize it to json, zip the json, and sending in chunks (multipart?), Are there guides to achieving this, sending zip in chunks to a server, i control everything (client/server) — ilansch, Aug 26 '13 at 14:48
+1 for this fact, sending json in chunks is possible ? Is there any benefit to serialize the class object to json before zipping and sending in chunks (zipped) ? are there alternatives ? In the server side i guess it will unzip the file, and deserialize to json. — ilansch, Aug 26 '13 at 15:12

score 5 · Accepted Answer · answered Aug 26 '13 at 16:12

On the client side this should work pretty much out of the box...

        var httpClient = new HttpClient();

        httpClient.DefaultRequestHeaders.TransferEncodingChunked = true;

        var content = new CompressedContent(new StreamContent(new FileStream("c:\\big-json-file.json",FileMode.Open)),"UTF8");

        var response = httpClient.PostAsync("http://example.org/", content).Result;

You can find an implementation of CompressedContent in WebApiContrib. If you are using earlier than .net 4.5 the request will be buffered client side before sending. Unfortunately the underlying HttpWebRequest doesn't support buffered streaming until .net 4.5

REST API Put large data

1 Answers1