Json.net Async when writing to File

Question

Json.net has the async functions for converting an object to json like:

json = await JsonConvert.DeserializeObjectAsync<T>

But when I want to write an object to a json file it seems better to me to do it directly using a file Stream.

So i think it should be something like this:

 var fileStream = await file.OpenAsync(FileAccessMode.ReadWrite);

    using (IOutputStream outputStream = fileStream.GetOutputStreamAt(0))
    {
        using (StreamWriter sw = new StreamWriter(fileStream.AsStreamForWrite()))
        {
            using (JsonWriter jw = new JsonTextWriter(sw))
            {
                jw.Formatting = Formatting.Indented;

                JsonSerializer serializer = new JsonSerializer();
                serializer.Serialize(jw, obj);
            }
        }

But on the JsonSerzializer Object I can't find async methods. Also I think that IO operations shouldn't be placed in a own thread.

What is the recommended approach ?

score 17 · Accepted Answer · edited Apr 08 '21 at 15:51

17

Json.NET doesn't really support asynchronous de-/serialization. The async methods on JsonConvert are just wrappers over the synchronous methods that run them on another thread (which is exactly what a library shouldn't do).

I think the best approach here would be to run the file access code on another thread. This won't give you the full advantages of async (it will waste a thread), but it won't block the UI thread.

edited Apr 08 '21 at 15:51

Collin K

15,277
1
27
22

answered Mar 26 '13 at 21:55

svick

236,525
50
385
514

Thanks for pointing that out. I was already wondering how a stringoperation could be non threaded asynchron.. The Streaming could be made asynchron. Perhaps that would be a Feature for Json.net but obviously it isn't implented yet. So Thanks for your answer. – Boas Enkler Mar 27 '13 at 06:49
This would be a simple task to complete - making an async fork of JSON.Net. The problem though, is that it would take a long time - anyone got about 80 hours to spare? if not more? – Kind Contributor Apr 24 '15 at 12:08
There is some support in the meantime: http://james.newtonking.com/archive/2017/03/21/json-net-10-0-release-1-async-performance-documentation-and-more But since the extension API offered by JSON.NET is not at all prepared for async (`JsonConverter` etc.), I think a fork would not make much sense. I guess a different serialization library specifically built with async in mind would make more sense. And async de-serialization would require a push-style parser (event-driven) instead of a blocking pull-style parser... – Lucero Jul 24 '18 at 13:52
@KindContributor Plenty of people. How do you think libraries such as that get written in the first place? Heck, a number of folks on github offered to help with this very thing over a span of years. But the author wasn't exactly supportive of the effort. And now he's essentially a ghost as far as Json.NET is concerned. – arkon Sep 22 '22 at 09:18
@arkon hence my comment that it could be forked, and it would be an easy task that an individual could accomplish. But these days (my comment was in 2015) it would be better to contribute to .Net Core. They have a better built-in JSON library now (I think it's based on Newtonsoft from memory) - and maybe that's the reason why there isn't much interest in volunteers putting hours in the original newtonsoft codebase – Kind Contributor Sep 28 '22 at 04:16

Manushin Igor · Answer 2 · 2018-07-24T14:22:56.617

9

See also this code, which uses right asynchronous way (e.g. it will not create huge byte arrays to avoid LOH memory allocations, it will not wait for IO operation complete).

// create this in the constructor, stream manages can be reused
// see details in this answer https://stackoverflow.com/a/42599288/185498
var streamManager = new RecyclableMemoryStreamManager();

using (var file = File.Open(destination, FileMode.Create))
{
    using (var memoryStream = streamManager.GetStream()) // RecyclableMemoryStream will be returned, it inherits MemoryStream, however prevents data allocation into the LOH
    {
        using (var writer = new StreamWriter(memoryStream))
        {
            var serializer = JsonSerializer.CreateDefault();

            serializer.Serialize(writer, data);

            await writer.FlushAsync().ConfigureAwait(false);

            memoryStream.Seek(0, SeekOrigin.Begin);

            await memoryStream.CopyToAsync(file).ConfigureAwait(false);
        }
    }

    await file.FlushAsync().ConfigureAwait(false);
}

Whole file: https://github.com/imanushin/AsyncIOComparison/blob/0e2527d5c00c2465e8fd2617ed8bcb1abb529436/IntermediateData/FileNames.cs

edited Jul 24 '18 at 14:22

answered Mar 09 '16 at 08:36

Manushin Igor

3,398
1
26
40

3

IMHO, this answer is better than the accepted answer, considering the accepted answer completely ignored the fact the user was attempting to perform IO and offers no real example. – James Haug Mar 22 '17 at 15:31
3

You pretend that your code "will not create huge byte arrays to avoid LOH memory allocations", but actually the `MemoryStream` internally does just that, it caches everything in a single byte array which is grown when needed. – Lucero Jul 24 '18 at 13:45
@Lucero, absolutely agree, we should use Microsoft.IO.RecyclableMemoryStream, will fix – Manushin Igor Jul 24 '18 at 14:17
3

@ManushinIgor Okay, that's better - even if it still requires the full data to be cached in memory. Note however that the link to the "Whole file" still has the old code. – Lucero Jul 24 '18 at 17:26
@Lucero, agree will fix on github a bit later – Manushin Igor Jul 25 '18 at 14:08

Stefan Steiger · Answer 3 · 2021-09-28T12:18:01.660

You cannot do that with JSON.NET/Newtonsoft.JSON.
However, you can instead use System.Text.Json.
To get the same behaviour as JSON.NET, just set IncludeFields and PropertyNameCaseInsensitive to true.

public static class JsonExtensions
{
    private static readonly System.Text.Json.JsonSerializerOptions _jsonOptions = 
        new System.Text.Json.JsonSerializerOptions
        {
            PropertyNameCaseInsensitive = true,
            IncludeFields = true
        }
    ;


    public static string ToJson<T>(this T obj)
    {
        return System.Text.Json.JsonSerializer.Serialize<T>(obj, _jsonOptions);
    }


    public static T FromJson<T>(this string json)
    {
        return System.Text.Json.JsonSerializer.Deserialize<T>(json, _jsonOptions);
    }


    public static async System.Threading.Tasks.Task ToJsonAsync<T>(this T obj, System.IO.Stream stream)
    {
        await System.Text.Json.JsonSerializer.SerializeAsync(stream, obj, typeof(T), _jsonOptions);
    }


    public static async System.Threading.Tasks.Task<T> FromJsonAsync<T>(this System.IO.Stream stream)
    {
        return await System.Text.Json.JsonSerializer.DeserializeAsync<T>(stream, _jsonOptions);
    }

}

and there you go.

Also, if you asynchronously want to serialize to string:

public static async System.Threading.Tasks.Task<string> ToJsonAsync<T>(this T obj)
{
    string ret = null;

    Microsoft.IO.RecyclableMemoryStreamManager streamManager = 
        new Microsoft.IO.RecyclableMemoryStreamManager();
    
    using (System.IO.MemoryStream ms = streamManager.GetStream())
    {
        await System.Text.Json.JsonSerializer.SerializeAsync(ms, obj, typeof(T), _jsonOptions);
        ms.Position = 0;

        using (System.IO.TextReader sr = new System.IO.StreamReader(ms, System.Text.Encoding.UTF8))
        {
            ret = await sr.ReadToEndAsync();
        }

    }

    return ret;
}

Json.net Async when writing to File

3 Answers3