Which kind of commpression do I have to use? Or use StreamContent with JsonTextWriter

Question

I'm developing an ASP.NET Web Api 2.2 application with .NET Framework 4.5, C# and Newtonsoft.Json 6.0.8.

I have this method to post to this web api:

protected bool Post<T>(string completeUri, ref T dataToPost)
{
    bool result = false;

    using (var client = new HttpClient())
    {
        client.BaseAddress = new Uri(_webApiHost);
        client.DefaultRequestHeaders.Accept.Clear();
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

        HttpContent content = new StringContent(JsonConvert.SerializeObject(dataToPost), Encoding.UTF8, "application/json");
        Task<HttpResponseMessage> response = client.PostAsync(completeUri, content);

        [ ... ]
    }

    return result;
}

When I have a lot of data to send I get an Out of Memory exception here:

HttpContent content = new StringContent(JsonConvert.SerializeObject(dataToPost), Encoding.UTF8, "application/json");

I have read a lot about compression but I don't know which one do I have to use. I don't know if there are more types but I have found two kinds: IIS compression and GZip compression.

Which one do I have to use? If I use GZip compression, do I have to modify my web api client?

UPDATE:

I have this class to serialize but I haven't used it:

public static string Serialize(Models.Aggregations aggregation)
{
    if (aggregation == null)
        throw new ArgumentNullException("aggregation");

    StringWriter sw = new StringWriter();
    JsonTextWriter writer = new JsonTextWriter(sw);

    writer.WriteStartObject();

    writer.WritePropertyName("Code");
    writer.WriteValue(aggregation.Code);

    if (!string.IsNullOrWhiteSpace(aggregation.Created))
    {
        writer.WritePropertyName("Created");
        writer.WriteValue(aggregation.Created);
    }

    writer.WriteEndObject();

    return sw.ToString();
}

It will solve the problem if I use it? I ask this because @CodeCaster has suggested me to use JsonTextWriter but I don't know how to use it inside my post method.

UPDATE 2
Following @CodeCaster recommendation I'm trying to optimize how I send data to that Web Api and I'm writing my own JSON serializer with this class:

public static string Serialize(Models.Aggregations aggregation)
{
    if (aggregation == null)
        throw new ArgumentNullException("aggregation");

    StringWriter sw = new StringWriter();
    JsonTextWriter writer = new JsonTextWriter(sw);

    writer.WriteStartObject();

    writer.WritePropertyName("Code");
    writer.WriteValue(aggregation.Code);

    if (!string.IsNullOrWhiteSpace(aggregation.Created))
    {
        writer.WritePropertyName("Created");
        writer.WriteValue(aggregation.Created);
    }

    writer.WriteEndObject();

    return sw.ToString();
}

But @CodeCaster has told me that to make it more efficient I will need to write as a Stream into StreamContent using JsonTextWriter.

But I don't know how to do that because I don't know how to instantiate StreamContent. All the examples that I've seen use a var stream but I don't see how they instantiate that object.

How can I use JsonTextWriter to write into the stream?

Compression will actually make your memory usage MUCH higher. You will need to allocate a buffer for the entire message (which you need to do now), PLUS a buffer for the compressed data. Assuming zero compression, you will end up using twice as much memory. — Aron, Dec 21 '15 at 09:43
@Aron only if you compress everything in-memory, you can also use streams, where the overhead will be just the buffer size. — CodeCaster, Dec 21 '15 at 09:49
@CodeCaster plus the buffer of your original stream. Which again roughly doubles your memory usage. — Aron, Dec 21 '15 at 09:50
@Aron no... if the serializer is streaming you won't have the entire JSON string in memory at once, and if the compressor is streaming then neither. Data will either be not serialized yet, in the process of being serialized or compressed, or compressed already and on the wire. Overall memory usage: lower than when not streaming. — CodeCaster, Dec 21 '15 at 09:52
@CodeCaster My point is that no matter what, adding compression to your stack increases the memory usage from the uncompressed usage. What you are talking about is a completely orthogonal change which requires structural changes to the code. — Aron, Dec 21 '15 at 10:02
@Aron I think so. The changes proposed CodeCaster changes everything on my code. — VansFannel, Jan 12 '16 at 08:41

score 0 · Answer 1 · edited May 23 '17 at 11:59

0

The compression you're talking about will not solve your problem.

It will only compress the data at the HTTP level. So in your code, you will still get the same, decompressed string, and the same OutOfMemoryException which occurs during deserialization.

You need to change how you deserialize the JSON. See Incremental JSON Parsing in C#.

Edit: sorry, I read entirely the wrong way around. It's the serialization, not deserialization. Still, same issue. Either the serialization itself, or the instantiation of the StringContent throws this exception. You need to serialize it streaming, and directly stream to the output.

So: use StreamContent, not StringContent, and use JsonTextWriter to write into the stream. This way you don't have to have the entire serialized string in memory.

edited May 23 '17 at 11:59

Community

1
1

answered Dec 21 '15 at 09:40

CodeCaster

147,647
23
218
272

1

Compression will actually make his problem worse. – Aron Dec 21 '15 at 09:41
But, the problem occurs when I serialize a list of objects. – VansFannel Dec 21 '15 at 09:42
I have updated my question with another class that uses `JsonTextWriter` but I don't know how to use it inside my post method. – VansFannel Dec 21 '15 at 10:34
@Vans the code you show will get you the same exception, as it's still serializing into a string. Serialize into a stream instead. – CodeCaster Dec 21 '15 at 10:35
Thanks. I'm searching but I don't know how to do it. I don't know how to use JsonTextWriter to serialize to a StreamContent. I have asked this question: http://stackoverflow.com/questions/34395355/use-jsontextwriter-to-write-into-streamcontent. – VansFannel Dec 21 '15 at 12:49
@Vans again, in that question you're showing a method that returns a string. That's never going to work. Please delete that other question and improve this one. – CodeCaster Dec 21 '15 at 12:55
Again, I don't know how to do it and I haven't found any sample on the web showing how to do that. This is why I haven't changed that class because I don't know how to do it. – VansFannel Dec 22 '15 at 07:09
@CodeCaster I have deleted that question and improve this one. – VansFannel Jan 12 '16 at 08:34

score 0 · Answer 2 · answered Aug 28 '18 at 09:38

Given results as the IEnumerable<T> containing your objects, you can use JsonTextWriter to write into the response OutputStream in the following way:

           // get the response output stream .. 
            using (StreamWriter stream = new StreamWriter(HttpContext.Current.Response.OutputStream))
            using (JsonTextWriter writer = new JsonTextWriter(stream))
            {
                // .. and use json.net to write the data as json in the stream  
                var serializer = new JsonSerializer();
                serializer.Serialize(writer, results);
            }

If you have your own JsonSerializer you can easily plug it in.

The response content will be compressed and chunked. Fast and neat

Response Headers
    Transfer-Encoding: chunked
    Content-Type: application/json
    Content-Encoding: gzip

Which kind of commpression do I have to use? Or use StreamContent with JsonTextWriter

2 Answers2