11

What is the difference between open-ended and close-ended load testing?
Thread-based load-testing tool comes in which category?
Which are some examples of load testing tools in each category?

Aliaksandr Belik
  • 12,725
  • 6
  • 64
  • 90
Stanly
  • 663
  • 1
  • 9
  • 26
  • 2
    I've been doing load testing for 17 years, and I never heard of the term "open ended load testing tool" or "close ended load testing tool". Can you reference some definition maybe? – timbre timbre May 12 '16 at 14:39
  • the terms may be misleading, but the concepts exists, have a look here: http://users.cms.caltech.edu/~adamw/papers/openvsclosed.pdf – Gerald Mücke Sep 20 '16 at 14:17

2 Answers2

14

Looks like you are speaking about closed vs. open workload models.

This classification based on closed/open systems separation:

  • in case of closed system model a new request is only triggered by the completion of a previous request, following by a think time. System has negative feedback that makes it impossible to bury-out the service, so users wait for the responses before making new requests;
  • in case of open system model new requests arrival independently of completions, e.g. according to a stochastic process or fixed trace. System has no negative feedback.

Based on this classification we can divide load-testing tools into following categories based on workload model used:

  • thread-based model where each thread will wait for a response before sending another request - so request rate depends on response rate (load generator drives the load as fast as system can take it):
    • thread-based approach is VU [virtual user]-oriented while trying to simulate N users working together;
    • thread-based tools require a separate thread to simulate a user;
    • examples: JMeter;
  • hit-based model
    • hit-based approach is RPS [request per second]-oriented while trying to produce N hits per second load;
    • hit-based tools use async approach and will always attempt to create defined request rate, queuing requests if necessary;
    • examples: tsung, Yandex.Tank.
Community
  • 1
  • 1
Aliaksandr Belik
  • 12,725
  • 6
  • 64
  • 90
  • 4
    I'll mention another open source tool: Gatling lets you generate load either way - both open and closed loops – Slavik May 07 '18 at 20:40
  • 3
    Actually, hit-based load can be generated by JMeter too, using [Throughput Shaping Timer](https://jmeter-plugins.org/wiki/ThroughputShapingTimer) – scormaq Feb 12 '20 at 11:57
10

I basically agree with the explanation of closed/open system from Aliaksandr and give you an example.

For example: you have 3 users, each iterating over a sequence of requests. Before a new iteration can start, the previous iteration has to finish. The previous iteration finishes, if all requests are finished. If the System-under-Test takes longer to respond, the request/s rate drops. So load-generator and system-under-test are in a closed loop.

In open loops, the request rate is constant, regardless of the response times - there is no feedback.

Both models reveal different performance characteristics of the system under test, i.e. capacity/throughput limits with closed-loop, queue sizes with open loops. It's more easy to overload a system with an open-loop

Regarding threads, usually thread-based tools define a close-loop, but, you could model open loops as well.

Take JMeter for example, if you want a constant rate of 2 users/s over a period of 3600 seconds, you use a rampup time of 3600 seconds and use a thread number of 7200, without looping.

Gatling is another free tool that is not thread-based but event based. You can model closed-loops (repeat, during) or open-loops (constantUsersPerSecs)

Gerald Mücke
  • 10,724
  • 2
  • 50
  • 67