2

I am trying to learn this new system.io.pipelines, and the new webapi strategy for deserializing json...

I wrote my own JsonConverter, but I can't figure out the correct way to initialize a Utf9JsonReader from a json flat file fixture.

here is the test:

    [Fact]
    public void WhenGivenJsonObjectThenEntityDTOReturned() {
                    
        using(var stream = new FileStream("Fixtures/BookStoreJson.json", FileMode.Open))
        {
            var pipe = PipeReader.Create(stream);
            ReadResult bytes;
            pipe.TryRead(out bytes);
            var reader = new Utf8JsonReader(bytes.Buffer);
            var target = new EntityDTOConverter();
            reader.Read();
            var actual = target.Read(ref reader, typeof(EntityDTO), new JsonSerializerOptions());
            
            Assert.True(actual.Props.ContainsKey("name"));
        }
    

    }

When I debug this, the bytes.buffer is set to 0 bytes, even though the BookStoreJson.json file contains the following:

{
    "name": "Tattered Cover",
    "store":{
       "book":[
          {
             "category":"reference",
             "author":"Nigel Rees",
             "title":"Sayings of the Century",
             "price":8.95
          },
          {
             "category":"fiction",
             "author":"Evelyn Waugh",
             "title":"Sword of Honour",
             "price":12.99
          },
          {
             "category":"fiction",
             "author":"J. R. R. Tolkien",
             "title":"The Lord of the Rings",
             "isbn":"0-395-19395-8",
             "price":22.99
          }
       ],
       "bicycle":{
          "color":"red",
          "price":19.95
       }
    }
 }
Nathan Tregillus
  • 6,006
  • 3
  • 52
  • 91

2 Answers2

4

Apologies,I didn't realise about async process, I was testing from console app. besides not sure how much this answer is going to help you out, basically you can run async tasks on sync by accessing results. also there is a limitation on buffer size, if the json file size is higher you may need to create custom pool and use AdvanceTo to options to read to end of buffer to get the stream.

using System;
using System.Buffers;
using System.IO;
using System.IO.Pipelines;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using Xunit;
 

            public class Person
            {
                public string Email { get; set; }
                public bool Active { get; set; }
                public string CreatedDate { get; set; }
                public string[] Roles { get; set; }
            }
    
            [Fact]
            public void WhenGivenJsonObjectThenEntityDTOReturned()
            {
                //{
                //    "Email": "james@example.com",
                //    "Active": true,
                //    "CreatedDate": "2013-01-20T00: 00: 00Z",
                //    "Roles": [
                //    "User",
                //    "Admin"
                //    ]
                //}
    
                using (var stream = new FileStream(@"c:\temp\json.json", FileMode.Open))
                {
                    var reader = PipeReader.Create(stream, 
                        new StreamPipeReaderOptions(bufferSize:4096 *2));
    
                    ValueTask<ReadResult> readResult = reader.ReadAsync();
                    ReadOnlySequence<byte> buffer = readResult.Result.Buffer;
                    Assert.True(readResult.IsCompleted);
                    var jsonStreamReader = new Utf8JsonReader(buffer);
                    var expectedJson = JsonSerializer
                        .Deserialize<Person>(Encoding.UTF8.GetString(buffer.ToArray()));
                    Assert.Equal("james@example.com", expectedJson.Email);
                }
            }
coder_b
  • 827
  • 6
  • 15
  • readResult is not defined just used as an out variable... is this a new C# thing? I'll give this a shot tonight! – Nathan Tregillus Dec 11 '20 at 19:16
  • this snippet just does not work: 1) the task must be marked `public async task` to use `await pipe.ReadAsync` 2) error CS4012: Parameters or locals of type 'Utf8JsonReader' cannot be declared in async methods or lambda expressions. which i didn't even know the compiler could fail in this way, but this and issue #3 is causing the answer above to not compile. 3) the variable readResult is not defined... These three problems block me from accepting this answer... – Nathan Tregillus Dec 12 '20 at 03:33
0

You can read a file by PipeReader in this way -

            using (var stream = new FileStream(@"test.json", FileMode.Open))
            {
                var pipeReader = System.IO.Pipelines.PipeReader.Create(stream);

                while (true)
                {
                    var pipeReadResult = await pipeReader.ReadAsync();
                    var buffer = pipeReadResult.Buffer;

                    try
                    {
                        //process data in buffer
                        Console.WriteLine(buffer.Length.ToString());

                        if (pipeReadResult.IsCompleted)
                        {
                            break;
                        }
                    }
                    finally
                    {
                        pipeReader.AdvanceTo(buffer.End);
                    }
                }
            }

If you set the buffer size for PipeReader larger than the size of the file to read, you do not need to loop, but that beats the purpose of using PipeReader - to process the data on the fly, piece by piece.

I do not think PipeReader is suitable to your case. Utf8JsonReader can directly work with a file stream. I think this is what you need -

https://learn.microsoft.com/en-us/dotnet/standard/serialization/write-custom-serializer-deserializer#read-from-a-stream-using-utf8jsonreader

ch_g
  • 1,394
  • 8
  • 12