2

We collect performance metrics like average response time, latency, Hits etc,. when we do load test. How can we evaluate the performance of a request using these metrics?

selenium
  • 55
  • 1
  • 10

2 Answers2

2

When you run performance tests, each test is run with some goal. It could be to baseline the application to assure that the application can handle certain amount of load without any SLA violations. SLAs are defined with product managers to set the performance expectations. These are basically in terms of acceptable response time ranges for APIs, throughput, page load time etc.

When you run your tests for the first time, you are creating a baseline for your application and creating load that is expected in real life. You collect metrics like response time, throughput to make sure that you are meeting SLAs defined for the product or how your application is performing under normal load conditions.

In case build to build performance tests, you need to make sure that throughput for latest and previous build is equal or more and response times have not deteriorated in subsequent builds.

If you running load tests with increasing load levels then response times, throughput can help you understand at what point your system is saturating and cannot scale further and what requests arent scaling well with increased load. In order to achieve this run tests with different load levels and compare response time and throughput for each request for under different load levels. This will give you clear picture of which requests are performing well under load and and which are not. Once you know the poorly performing requests, run tests for that particular request and add code profilers along with the load tests to find the bottlenecks.

rachna bafna
  • 501
  • 3
  • 7
  • Thanks for the info. Very helpful. Can you say further in terms of memory, CPU utilization,.. – selenium Mar 08 '18 at 13:33
  • Make sure your CPU doesn't go beyond 80% when running load tests, also check the number of times GC is kicked in and time to complete GC. Each system has different number of issues associated with them in terms of performance. Need to start with memory utilization and GC to begin with and profile the code to dig deeper. – rachna bafna Mar 15 '18 at 18:41
  • Is the the responsibility of the performance tester to do code profiling? – selenium Mar 17 '18 at 13:03
  • Yes, Ideally performance engineers profile the code, find bottlenecks and work with developers to get them fixed. – rachna bafna Mar 18 '18 at 16:51
2

Response time is only telling if you have monitored resources to match it to. Hits may or may not be telling, as you can shape hits by the management of your cache plan. But, ultimately, what are your requirements from the business stakeholders asking you to measure for success? If you cannot meet their measure in your test then you need to examine why.

The value of the test is not the test itself, but the collection of timing records and matched resource measurements leads to analysis which leads to value.

If you are not monitoring, then you are not engaging in a performance engineering activity. You are just throwing load. Performance Testing is a performance engineering activity.

James Pulley
  • 5,606
  • 1
  • 14
  • 14
  • Thanks for the info. How can you shape hits by the management of cache plan? – selenium Mar 08 '18 at 13:28
  • Example. Font resource on every page. Fonts don't change that often, so if you have no plan, no etag, then you may have a hit for a request on every page. If your cache age is not long enough then you will have a hit on HTTP 304, unmodified. If you have a cache of your session duration, then one request per user per session. If you have return users and you cache across days, then one hit per user per multiple days. If in CDN with a policy of one year, then one hit per CDN node for the next year for the font. This is HTTP architecture stuff – James Pulley Mar 08 '18 at 16:23
  • If you simply want to measure every single request, not a logical grouping such as a page, then just turn on the w3c time-taken field in your logs and process the logs with tools such as Microsoft LogParser, Splunk, SumoLogic, the ELK stack, etc... – James Pulley Mar 08 '18 at 16:52