0

We have one old ASP.Net asmx webservice in our application which receives bulk requests at sometime. Service is taking less than 5 seconds for a single request. But It is taking more than a minute when it receives 20 or more concurrent requests. Following is the way it is implemented,

1)receives a request with input data from external clients

2)Will get 20 possibilities from database for one request based on input data after validation

3)Then It will iterate all 20 possibilities using foreach and gets solutions either from other external service or data base based on possibility data.  Here in old implementation we have used Parallel.Foreach to perform all 20 calls (service calls or DB calls) parallely to improve the performance.

4)After that Service will send back the all 20 solutions to the client.

This old approach is working fine for few (1or 2 ) requests and resonse time of asmx service is very fast(less than 5 seconds)  considering external service calls which are taking 2-3 seconds .But This approach is taking more than 60 seconds when the number of concurrent requests are more  than 20.Concurrent requests are pushing CPU utilization to 100% and thread pool starvation as per experts analysis and there by causing requests to queue for threads allocation.

So we got a recommendation to replace parallel extensions and complete service with async/await implementation from end to end.I have implemented async/await end to end and also replaced Parallel.foreach with Task.WhenAll in TPL. But response time has increased a lot after this implementation.for a single request 20 secconds and it its taking more than 2 minutes for bulk requests.

I also tried async foreach in place of parallel.foreach as mentioned in below article  but still performance is really bad.

https://stackoverflow.com/questions/14673728/run-async-method-8-times-in-parallel/14674239#14674239 

As per logs basic issue is with external service calls/DB calls inside foreach in both old parallel or new async/await implementations.But these service responses are very fast for a single request. Async implementation is taking more time in completing service calls than parallel extensions implementation.

I think service should not take more than 20 seconds for bulk request if it is lessa than  5 seconds for single request.

Can anyone please me what should be the way forward here to improve the performance ?

Thanks in advance.

Regards,

Raghu.

Raghunath Machineni
  • 43
  • 1
  • 6
  • You could consider adding a caching layer between your app and the database, assuming that caching some frequently requested data is compatible with your app's functionality. – Theodor Zoulias Nov 10 '20 at 10:17

1 Answers1

2

Looks like a lot of things happening here at the same time. I believe you have on nderlying issue that causes many side effects.

I will make the assumption that your server is sufficient in terms of CPU and memory to handle the concurrent connections (though the CPU 100% makes me wonder).

It seems to me that your problem, is that the parallel tasks (or threads), compete for the same resources. That would explain why multiple requests take much more time and why the async paradigm takes even more.

Let me explain:

The problem in practice

Parallel implementation: 1 or 2 request need minimum synchronization, so even if they compete for the same resources, it should be fine.

When 20 threads, try to access the same resources, a lot is happening and you come to a situation known as livelock.

When you switch to async, no requests await for a thread (they are waiting on the IO threads), so you make the problem even worse.

(I suspect that the problem is on your database. If your database server is the same machine, it would also explain the utilization).

The solution

Instead of trying to up the parallelism, find the contested resources and identify the problem.

If it's in your database (most probable scenario), then you need to identify the queries causing the trouble and fix them (indexes, statistics, query plans and whatnot). DB profilers showing locks and query execution plans are your friends for this.

If the problem is in your code, try to minimize the race conditions and imporve your algorithms.

To get a hint of where to look for, use the Visual Studio profiling tools: https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019 or any external .net profiling software.

Athanasios Kataras
  • 25,191
  • 4
  • 32
  • 61
  • Thank you so much for your response, I could see that contested resources are mainly external service calls and database call as per logs. We need to make 3-5 external service calls and one database stored proc call for each iteration. All these calls are taking maximum time of processing which is around 2 minutes for async and 1 minute for parallel extension. But all these external service calls and stored proc call are working very fast for single request. – Raghunath Machineni Nov 11 '20 at 06:13
  • We also improved DB logic by indexing wherever needed. So I thought there is no issue with the actual business logic in external service or stored proc.Also rest of my service implementation has no other issues apart from these forach or parallel for each iterations what could be the options in this case to improve performance at my service end ? Or Is it due to external service /DB not able to handle multiple requests ? Thank you – Raghunath Machineni Nov 11 '20 at 06:19
  • You need to drill down to the problem. Imagine a store proc that locks some database rows in an ambient transaction. Now imagine that this one runs first and then calls a service. The service takes 2 minutes and the db rows are still locked. No other call that needs the same rows can be executed, and are blocked on the db level. – Athanasios Kataras Nov 11 '20 at 06:41
  • Databases also have profilers that can inform you of the blocks happening there. – Athanasios Kataras Nov 11 '20 at 06:41
  • Our database stored procedure contains very simple select statement with out any further business logic.I can not question external service vendors since their service is very fast for single request. Is there a possiblity that external service is not performing when number of requests are more ? I am definitely going to run profiler for application and DB as you mentioned. Thank you so much – Raghunath Machineni Nov 11 '20 at 09:40
  • I have done profiling for my service and it is showing me 80% of CPU is utilized for external I/O Calls for single request itself. There are 5 I/O calls to external services.So can I say now that issue is with external service response ? – Raghunath Machineni Nov 13 '20 at 10:36