26

I am doing some profiling on a WCF service that uses EF (System.Data.Entities) to read from a SQL DB. When I spin up multiple parallel clients that hit the service, the CPUs all go to 100%, performance generally tanks, and everything bogs down.

In profiling this with the concurrency profiler, I found 85% of the time is spent in synchronization, with only about 4% being actual code execution. Looking deeper into the stack trace, most of the synchronization seems to be from a call to WaitForSingleObject in System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync. The stack shows that the call goes to a native method wrapper and then winds up at kernel32.dll!_WaitForSingleObject.

Has anyone experienced this before? Is there any way to do something about this? I'm not really throwing absurd load at this, only about 20 parallel clients, and it's all read-only, so I'm surprised the threads would even bother to synchronize.

I've been fighting with this for a week now and I just can't explain it. Any help would be appreciated!

Osama Rizwan
  • 615
  • 1
  • 7
  • 19
XeroxDucati
  • 5,130
  • 2
  • 37
  • 66
  • Could the client be waiting for the server to reply? – usr Sep 29 '12 at 15:32
  • 1
    The client definitely is waiting, but it's running on a different machine.. this profiling is coming from the server, where nothing should be waiting. SQL is offloaded to another machine as well, and it's barely registering on cpu usage (1%) and network (<1%). The data is returning in under 100ms according to sql profiler. – XeroxDucati Sep 29 '12 at 15:42
  • "85% of the time is spent in synchronization": CPU time is not spent there. This wait is not using CPU time. What are the threads doing which *are* burning CPU? You say the CPU is ~100% busy. Also please show longer stacks because many path can lead to WaitForSingleObject and such leaf methods. – usr Sep 29 '12 at 17:41
  • 2
    Thats the weird thing, the threads don't actually seem to be doing anything at all, they're all sitting in 'synchronization'. I ran debugdiag as well, top 10 threads by max CPU all show the same thing: – XeroxDucati Sep 30 '12 at 13:26
  • async is defaulted, so it's true. I don't have any synchronous calls unless EF is creating some for some reason.. I have not spotted any. – XeroxDucati Oct 01 '12 at 02:17
  • 1
    Well that is clearly sync. There are no callbacks and no tasks. So it is sync (which is not wrong at all). You can choose sync or async freely, but you have to be consistent. I suggest you stick to sync. – usr Oct 01 '12 at 15:23
  • 1
    I suspect that `SNIReadAsyncOverAsync()` is called by ADO.NET in a busy loop when SQL Server sends a partial response and then takes a non-zero amount of time to send a subsequent response. I think that it takes the fact that it got row data as a hint that more row data should be read synchronously to fill some buffer it has even though SQL Server might not provide data for a long time. I myself [am having trouble getting `SqlDataReader.ReadAsync()` to reliably run asynchronously](https://stackoverflow.com/q/45597995/429091) because of this. Maybe it is busy looping instead of blocking for you – binki Oct 06 '17 at 18:53

1 Answers1

3

Are you able to distill this to a small code sample that reproduces the problem? What version of EF are you using?

Here are a few observations based on the information you've given so far.

EF Async

Anything less than EF 6 is always synchronous. With EF 6 you have the option to use async methods instead. However, don't do this unless your WCF service also uses the async pattern.

WCF Async

You can write a WCF service whose implementation is asynchronous. See this documentation for more information.

If you use one of the above methods, but not both, your code will not be asynchronous but will incur unnecessary synchronization overhead. Especially avoid Task.Run() or equivalents, since these will simply move work to another thread without actually improving throughput.

Initialization

Finally, another unrelated idea. Could your issue be related to EF initialization? When EF builds the metadata for a model it does this once per connection string. If multiple threads attempt to use the same model and that model has not yet been initialized, all threads will block until initialization is complete. To see if this is your issue, make one call to the service and allow it to complete. Then submit your 20 parallel requests. Do they still max out the CPU?

Olly
  • 5,966
  • 31
  • 60