2

Disclaimer: Since I am unsure of what the issue really is, I have put it into Stackoverflow instead of Serverfault.

Description I am an owner of an application that approaches launch, so I made a tool that does regressiontests and stresstesting (loadtesting), when we're testing with 2-3 clients we see no impact, but as soon as we reach 8-10 clients we see huge impact in service delays and handle time from our API.

TPS = Tests Per Second initiated by my tool (clients/threads initiated every second)

Here is the output from my stress test tool:

  • TPS @ 5 - Avg. handle time: 1302 ms
  • TPS @ 10 - Avg. handle time: 5641 ms
  • TPS @ 30 - Avg. handle time: 13549 ms
  • TPS @ 50 - Avg. handle time: 6136 ms
  • TPS @ 100 - Avg. handle time: 24854 ms

Notes:

  • A. What I usually see, is no real pattern between the TPS, except it just takes long time after 10+ TPS. As you can see 50 TPS is much faster than 30 TPS.
  • B. What I also noticed, is that it looks like it queues requests, and there is some time between it completes new requests, see this screenshot: https://gyazo.com/1431b5113ac216983a6ca6e1f1bd75ad
  • C. The only thing I saw improve, is when removing external calls in the code and running the code again (external API), but when testing the external API isolated E2E, I see no delay (5000+ TPS with 120 ms avg.), but we can't handle more than 10 TPS without delay through our service.
  • D. No hardware graph ever gets over 5% (memory or CPU)

Do you have any suggestions, where should I look next? I am open to questions.

Technology:

  • C# (.NET Core) IIS WebService

What I have done/tested (with no impact):

  • End to end stres-testing the external applications we use, here we can easily handle 5000+ TPS with no real delay.
  • Implemented LoadBalancers (NGINX) on all layers (front-end, service, integration)
  • Used MongoDB Optimizer to attempt to speed up data flow.
  • We have upgraded our DB from 16 GB -> 64 GB and 2 CPU -> 8 CPU
  • Looked through IIS setup, to see if I can find something that looks odd.
  • Deactivated Antivirus, to see if external HTTPS calls are impacted.
Zwac
  • 67
  • 1
  • 5
  • Possible duplicates: - https://stackoverflow.com/questions/52042077/concurrent-request-in-asp-net-web-api-2-is-nearly-six-times-slower-than-a-single - https://stackoverflow.com/questions/28543293/entity-framework-async-operation-takes-ten-times-as-long-to-complete Looked through both questions, without any luck. – Zwac Jun 03 '22 at 12:09
  • I'm not trying to advertise for anything, but our IT department recently did a trial run with Lean Sentry on an application that was hanging and queueing requests in the IIS worker process. It was able to point out an issue with a piece of code that was actually in the .Net Core package for `SQLClient` and the issue was `ConnectAsync` from ADO.Net. Would I have found this myself? Probably not, unless I got really good at reading the IIS log reports. You can use it for free for 1 week. It might be enough to shed some light on if there is some blocking code in your application. – Ryan Wilson Jun 03 '22 at 12:21
  • If you have C# as a tag, you need to look at the applications/web services which are actually running. If they do not offload "busy waits" to the system by strictly enforcing any long running call as `async`/`await`, it definitely slows down IIS because the items are not releasing the limited resources while waiting, say for a database call. – ΩmegaMan Jun 03 '22 at 12:29
  • @RyanWilson I saw this tool a couple of days ago, I will look into this - thank you. – Zwac Jun 03 '22 at 12:36
  • @ΩmegaMan I have been worried regarding resource locking in the database as well as non-ideal connectionflushing, I will look further into this - thanks – Zwac Jun 03 '22 at 12:36
  • I had a commercial product release which was ultimately slated for Black Friday operations. The webservices/database had to handle thousands of transactions. In the sections of code (EF I'm looking at you) which didn't do the async/await paradigm, definitely failed our initial stress tests to that endpoint. It was my job to go in and async/await the db calls. GL – ΩmegaMan Jun 03 '22 at 14:16
  • Learn a good enough performance profiler and then use it to analyze the app when it is slow. Other people won't be able to help much as you are the only that can touch the environment. – Lex Li Jun 03 '22 at 16:54

0 Answers0