How to write asynchronously using CsvHelper in .Net 6

Question

Is it possible to write asynchronously using CsvHelper 30.0.0 in .Net 6.0?

The WriteRecord(s) method consistently throws this error:

Microsoft.AspNetCore.Server.Kestrel.Core: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.

Here is the first approach I tried (which I got from https://stackoverflow.com/a/55928375/18463829):

foreach (var item in items )
  {
    csvWriter.WriteRecord(item);
    await csvWriter.NextRecordAsync();
  }

I also tried this:

await csvWriter.WriteRecordsAsync(items);

But that has the same result.

In the code above, "items" is a List of simple DTO's. We have many DTO's but all of them are simple objects, and they all generate the same error. Here is a sample DTO (with a couple of names changed):

using System;

namespace CompanyName.Domain.DTO
{
    public class SomeDTO
    {
        public DateTimeOffset StartDateEST { get; set; }
        public int Value { get; set; }
        public long SomeID { get; set; }
        public short SomeLevel { get; set; }
    }
}

Additional background info:

This is a .Net 6.0 Azure Functions app, targeting Azure Functions v4, and using CsvHelper 30.0.0. It is a set of APIs, each with multiple operations. It is an old app, and this issue was introduced when we upgraded to .Net 6.0. All of the endpoints work as expected when the consumer requests the data in JSON format, but when they request CSV, we have the issue described above.

When deployed to Azure, there is no error thrown. There is nothing in the logs. The endpoints return a 200 "ok" response, but with no data (even though there is lots of data, and the same request returns the data if the consumer requests it in JSON).

When I run it locally, I can see the above error message in the console. I can also see this in the VS Output window:

Exception thrown: 'System.InvalidOperationException' in Microsoft.AspNetCore.Server.Kestrel.Core.dll
Exception thrown: 'System.InvalidOperationException' in Microsoft.AspNetCore.Server.Kestrel.Core.dll
Exception thrown: 'System.InvalidOperationException' in System.Private.CoreLib.dll
Exception thrown: 'System.InvalidOperationException' in System.Private.CoreLib.dll

However, I cannot seem to get Visual Studio to capture the stack trace. I've been searching for some kind of setting that I'm missing, but no luck. It's as if Visual Studio is unaware that the error is happening.

Try to make the first line in the foreach loop like this: `await csvWriter.WriteRecordAsync(item);` And leave the rest of the code like this. Please try this out and tell me if it fixed your issue. — AztecCodes, Jun 01 '23 at 13:58
What's the *full* exception text? What are you trying to do? Which ASP.NET Core version are you using, what environment? `WriteRecord` fills the output buffer, it doesn't write to the output stream. You could replace the loop with `WriteRecordsAsync` but that uses the same loop too. I suspect the error is throwne somewhere else, or is due to a deprecated, fixed behavior. Googling that message returns results from the discontinued ASP.NET Core. The oldest supported .NET Core version is .NET 6 — Panagiotis Kanavos, Jun 01 '23 at 14:03
@PanagiotisKanavos Perfectly reasonable error to get if the stream used by the writer doesn't allow synchronous. Why did you delete your answer, it seems to be correct, looks like that function got added some time ago https://github.com/JoshClose/CsvHelper/issues/1377 — Charlieface, Jun 01 '23 at 14:18
@Charlieface which is a rare case and appears only in search results associated with ASP.NET Core 3. I deleted the answer because `WriteRecordsAsync` [also uses `WriteRecord` to generate the output buffer](https://github.com/JoshClose/CsvHelper/blob/7b3ed4d45af8385e732a42eb161b0d129736edb3/src/CsvHelper/CsvWriter.cs#L458). WriteRecord *shouldn't* be trying to write to the output stream, that's done by [FlushBufferAsync](https://github.com/JoshClose/CsvHelper/blob/7b3ed4d45af8385e732a42eb161b0d129736edb3/src/CsvHelper/CsvWriter.cs#L513). — Panagiotis Kanavos, Jun 01 '23 at 14:24
This means something else is wrong, that may be affected by the CsvHelper configuration or version. CsvHelper publishes a major version every other month, introducing breaking changes quite frequently. We need the full exception call stack to see which method actually threw and why. — Panagiotis Kanavos, Jun 01 '23 at 14:27
Digging in GitHub shows `WriteRecord` ends up calling one of 3 possible record factories. Perhaps one of them is actually trying to write to a stream when it shouldn't? — Panagiotis Kanavos, Jun 01 '23 at 14:28
1) What version of CsvHelper are you using? 2) Can you please [edit] your question to share the full `ToString()` output of the exception including the exception type, message, traceback and inner exception(s), if any? 3) Can you please share a [mcve] that includes all of your CsvHelper calls? For instance, how do you construct and dispose of the Stream and TextWriter? — dbc, Jun 01 '23 at 17:15
@PanagiotisKanavos I tried WriteRecordsAsync, and as you suggest, got the same result. I updated the description with more info, and am trying to capture the stack trace now. — Jim Robson, Jun 01 '23 at 17:36
@JimRobson - I tried to reproduce your problem using a custom `SurrogateStream` that throws on any synchronous calls and I cannot, see https://dotnetfiddle.net/G8cH7L. So, as explained in [ask], since you are trying to fix a problem with your current code, we need to see a [mcve], showing all your CsvHelper calls as well as the input `item` type. — dbc, Jun 01 '23 at 17:38
Incidentally, in your edit you mentioned translating from JSON to CSV, did you switch from Newtonsoft to System.Text.Json? — dbc, Jun 01 '23 at 17:42
Then please share the code that does that. If I try to serialize a `List` to CSV using CsvWriter I get an `InvalidOperationException`, see https://dotnetfiddle.net/ubEel1. `List` does work though, see https://dotnetfiddle.net/t4dwlx. — dbc, Jun 01 '23 at 17:54
@dbc I think you are on to something. I see this in the VS Output window: Exception thrown: 'System.InvalidOperationException' in Microsoft.AspNetCore.Server.Kestrel.Core.dll Exception thrown: 'System.InvalidOperationException' in Microsoft.AspNetCore.Server.Kestrel.Core.dll Exception thrown: 'System.InvalidOperationException' in System.Private.CoreLib.dll Exception thrown: 'System.InvalidOperationException' in System.Private.CoreLib.dll — Jim Robson, Jun 01 '23 at 18:03
Still no [mcve] I'm afraid. I wrote some entirely asynchronous code to transform from JSON to CSV using Newtonsoft here: https://dotnetfiddle.net/WbdIEO. Is this what you want? Note you might need to add calls to `ConfigureAwait(false)` to make everything work on kestrel, see [Async processing in asp.net core and kestrel thread pool](https://stackoverflow.com/q/41101797). — dbc, Jun 01 '23 at 18:35
Azure Functions is a special environment with special limitations. As for `we upgraded to .Net 6.0` which version did you migrate from? There *was* such a change in ASP.NET Core *3*, not 6. It's well documented, there were some issues when 3.0 was introduced and there's a [compatibility setting](https://github.com/Azure/azure-functions-host/issues/7064#issuecomment-7710206450) to force Azure Functions to work the old way while you migrate your code. — Panagiotis Kanavos, Jun 02 '23 at 06:45

score 1 · Answer 1 · answered Jun 01 '23 at 19:34

1

After looking at the code samples provided by @dbc, I realized the existing code was missing a couple of "await" keywords. For example, it had:

using (csvWriter)

Where it should have had:

await using (csvWriter)

I added them in, and now it works fine.

FWIW, before I posted this question, I searched for CsvHelper documentation, but did not find any. Instead, what I found was "Coming soon..." Based on the StackOverflow answer referenced in the question, I thought there was no such method as csvWriter.WriteRecordsAsync(). Since that answer did not work for me, and it is a few years old, I figured it was out-of-date, and hence posted this one.

answered Jun 01 '23 at 19:34

Jim Robson

23
5

That means the error didn't even come from the code you posted. Had you posted the exception people asked for, that would be obvious. This isn't a CsvHelper issue at all. The problem appeared in ASP.NET Core *3.0*, not 6. It's well documented in the 3.0 release notes, there are Github Issues about this in the [Azure Functions site](https://github.com/Azure/azure-functions-host/issues/7064#issuecomment-771020645) and some compatibility settings you can set – Panagiotis Kanavos Jun 02 '23 at 06:48
`I cannot seem to get Visual Studio to capture the stack trace.` you don't have to. It's part of the exception itself. All you need to do is log the output of `Exception.ToString()` – Panagiotis Kanavos Jun 02 '23 at 06:49
@PanagiotisKanavos I am sorry for wasting your time. Thank you for trying to help. – Jim Robson Jun 02 '23 at 14:25

How to write asynchronously using CsvHelper in .Net 6

Additional background info:

1 Answers1