Drop in Jmeter hits per second after 2 hours

Question

There is a drop in hits per second in our load tests from jmeter after about 2 hours. Jmeter is running non gui in distributed fashion. My thread count per minute is 25k and we run with 6 servers.

Inorder to rule out the jmeter as the bottleneck , i checked if Jmeter has a connection pool with a size limit.

But Jmeter seems to maintain http connection pools per thread. As per Does JMeter pool HTTP connections?.

Its clear that the default value is 2000 milliseconds as per httpclient4.time_to_live property in jmeter.properties

# TTL (in Milliseconds) represents an absolute value.
# No matter what, the connection will not be re-used beyond its TTL.
#httpclient4.time_to_live=2000

Version of jmeter: 4.0.0 which uses Http4 client implementation

This means Jmeter creates connections for each thread which times out in 2 seconds.

The receiving jetty web server has following connection pool values:

maxThreads = 3200
minthread = 656
idleTimeout = 60000

Is it possible that the drop in hits per second from jmeter is induced due to the slow / no response from jetty server.? Are there any rules around matching the Jmeter thread count with the target web server's connection pool. ?

Note: I understand that the throughput generally depends on the response from the Application Under Test. But will Jmeter's hits per second to the target server be in any way be affected by the Application under test.?

Update: From https://stackoverflow.com/a/40689714/1165859, its clear that there is a correlation between Hits per second and Application Under Test. But my issue is that everythings fine for the first 2 hours. After which the Hits per second drops.

With these kinds of numbers, you are testing on different machines (1 server machine, and about 5 client machines) right? — Joakim Erdfelt, Oct 04 '19 at 15:18

score 0 · Answer 1 · answered Oct 08 '19 at 13:14

What you're doing is more like to be a Soak Testing which main goal is checking for memory leaks and given throughput degradation over time it looks like to be a "classic" memory leak.

So my recommendations are in:

Use JVisualVM or equivalent to monitor heap usage, GC activity, threads behaviour, etc. on Jetty side
Use JMeter PerfMon Plugin to monitor CPU, Memory, Swap, Network, Disk usage, etc
Use a profiler tool like JProfiler or YourKit to look for largest objects, "heaviest" methods, etc.
You can also apply the above 3 points to JMeter

Drop in Jmeter hits per second after 2 hours

1 Answers1