-1

There is a need to find a performance bottleneck in server application under big load. Application consists of single services instance (.asmx) and some files that are requested over http from time to time. My plan to solve this problem is 1) get to exceptional situation when server starts failing somehow 2) analyze performance counters and logs in that moment of time to deduct what kind of calls caused that.

To start achieving this I've implemented a special client that issues both types of requests and made it repeat respective cycles indefinitely hoping at some point I'll get errors during WebMethod/GET url requests (NB - standard already existing solutions like JMeter and WAPT can't be used duo to complexity of services usage scenario). So far what I am observing is increased response time in service calls and some network timeout exceptions during files loading (using HttpClient that throws OperationCanceledException which is considered timeout according to - this thread). Btw, that's strange, because files are few kb in size, and service methods returns 5-10 mb of data per request. Thought "larger" requests are more likely to fail first.
Perfmon shows increased CPU load and absolutely no memory spikes/leaks. Request Execution Time counters are pretty random and looks irrelevant, Queue Lengths are always 0.
That said, looks like IIS handles my improvised DDoS well and at the same time makes testing approach ineffective (increased response times means more active requests in memory on test client which causes memory overflow at some point, and I'm already flushing data right after I receive it without doing anything with it).
More details : server machine is 4x3Ghz cores, 4 Gb RAM. I generate load of 50-100 requests per second which results in 10-20 Mb/sec bandwidth (test clients are situated on VM inside server's datacenter, 4 Gbps NIC). 30 minute testing session is ~10-30 Gb of pure data transfer between server and client.
How can I actually make Web Service/IIS go down?

Community
  • 1
  • 1
Jaded
  • 1,802
  • 6
  • 25
  • 38
  • I was looking for a way to achieve some failure in server infrastructure and find performance bottleneck, but now it's clear that brute-force approach doesn't work. Please delete this question. Thanks in advance. – Jaded Sep 21 '13 at 14:16

1 Answers1

1

Firstly, I wouldn't write my own load testing tool; there are plenty available. I've used JMeter (open source). You can use JMeter (and other similar tools) to send both POST and GET parameters, cookies and other HTTP headers - though admittedly, this does become challenging for complex cases.

Next, make sure your problem really is the server, and not the other infrastructure - network, routers, firewalls etc. all have maximum capabilities, and may be the root cause of the problem. Most of them have logging and reporting tools. For instance, I've seen tests report a throughput issue when they reached the maximum capacity of the firewall; the servers were not even close to breaking point. This happened because we had included a rather large binary file in the test cases, which normally would be served from a CDN.

Next, on the whole it's unlikely that serving static HTTP requests is the problem - IIS is really, really good at that. For the kind of hardware you mention, I'd expect to handle many thousands of requests per second. for static files.

In most situations, it's the dynamic pages that cause the problem - your .asmx. So, I'd ignore all the static files in the load testing, and focus on the .asmx. On the kind of hardware you mention, you probably need to generate many hundreds of requests per second if the asmxes are working properly.

Working on the assumption that your web server is tuned correctly, and the asmx scripts are reasonably performant, I'd expect to need at least twice the (CPU and memory) capacity from the test system as your server has to bring it to breaking point (this is based on my experience with JMeter, which is not as efficient as my web applications, but does make it easy to deploy multiple test clients). So in your case, I'd look for 2 machines matching your server specification.

With JMeter (and pretty much all the other load testing tools I've worked with), you can fairly easily use multiple machines as load test clients; I've also used Cloud-based load generation using JMeter.

I'm not totally sure why this rule of thumb is true - but I've observed it over multiple projects.

Neville Kuyt
  • 29,247
  • 1
  • 37
  • 52
  • 1) Tried that multiple times, not applicable. There are a lot of dynamic parameters passed to service, so all tools like JMeter and WAPT doesn't simulate client activity and eventually hit SQL Server cache and produce minimal load. 2) Current test machine is on VM in same datacenter as server and has 4 Gbps NIC. High degree of probability that network is not an issue. 3) That's exactly what I'm thinking of. Where do I find some numbers/statistics to prove that? Thanks for help btw, doesn't understand why this issue may be voted to close. Performance testing is very interesting science. – Jaded Aug 31 '13 at 07:48
  • You might want to update your question with those additional facts - it helps others who might want to provide answers. I've updated my answer in response. People are voting to close because your question appears to invite opinion or debate, rather than a factual answer, not because the topic isn't interesting. – Neville Kuyt Aug 31 '13 at 10:02
  • Thanks again for provided information. Could you please clarify the last statement more? Did you mean "to break 4-core 4Gb server we need test client machine that has at least twice more resources"? I suspected something like this, because when I do testing, at some point quantity of requests decreases duo to increasing response time, memory usage grows exponentially and I start thinking test workstation messes testing. Yet not sure how to overcome this, because when server responds longer, a lot of service handles are waiting for response in memory and nothing can be done with that. – Jaded Aug 31 '13 at 15:12
  • Yes, that's what I meant - though usually, it's easier to use multiple machines to run the tests, rather than one single bigger machine. – Neville Kuyt Sep 01 '13 at 18:30