-1

I have been working on an update of the YottaDB web framework benchmarks. I am having trouble working out the difference between locust request times and response times. This answer implies that response time includes all requests to that endpoint.

But these benchmarks were created by sending >10,000 requests to just one endpoint. Does that mean response time should be identical to the request time? Or should the response time cover all 10,000 requests? It seems to be neither.

My results summarised here show for the Lua web stack, an average request time of 1422ms but a 50%tile response time of 250ms. I you want more detail, here's the benchmarking setup.

How are these two related?

Berwyn
  • 151
  • 7
  • 1
    Can you clarify what you mean by "request time"? – Solowalker Aug 25 '22 at 12:44
  • No, I don't know exactly what Locust means by "Request time". That is precisely the question I'm asking: specifically what is being measured by each of these metrics? – Berwyn Aug 26 '22 at 22:36

1 Answers1

0

"Response time" is the amount of time between your user making the initial request to when a response from the server is received. Locust's http client inherits from Requests so you can research more of what that response time (sometimes called elapsed in code) means technically, but the gist is that is how long the use is waiting for a response. That includes network time but is primarily determined by the amount of time it takes your server to accept the request, do whatever work it needs to, and then send the response.

EDIT:

Your specific concerns about why there's a difference between the average and 50%ile response times comes down to what average and percentiles are. They are not the same thing so the numbers are expected to almost always be different (unless you have a very responsive system that you're not able to stress sufficiently and all your response times come back as roughly the same).

Here's some simplified math to help you understand using your specific data. In a comment, you mentioned yottalua's data so we'll use that. yottalua had 21781 requests with an average response time of 1217ms. We get that by adding up the response time for every single request and divide it by the number of requests, meaning the sum total of all response times for all requests must be around 21781*1217=26,507,477ms.

The %tiles, though, give you more detailed information. They are saying that a large majority—somewhere between 80% and 90%—of your requests are fine with low response times, but things start to break down from there. 10%-20% of the requests have a really high response time. With this information we can approximate an average to compare with what Locust has already told us the average is. Let's pick some numbers and say that 15% (the middle between 10% and 20%) of your requests (21781*.15=3267.15 requests) have high response times and say they're all 6000ms (a number close to the middle between 2400ms that's your 80%ile and 13000ms at your 100%ile) and all the other 85% of your requests (21781 *.85=18513.85 requests) had 240ms (60%ile).

((21781*.15*6000)+(21781*.85*240))/21781 = 1104

Again, this is simplified and was arrived at with picking random numbers to make the math easier, but it gets us in the ballpark of the reported average (1104ms vs 1217ms).

You may have a majority of requests with a low response time (e.g. 80% of 21781, or 17,424.8 requests with 290ms) but your minority of requests with a really high response time (e.g. 20% of 21781, or 4,356 requests with 2400ms-13000ms) will skew the average higher than you think. But both your percentiles and averages will be correct even though they are different, they're just different ways of looking at the body of data you have that is your test results.

Solowalker
  • 2,431
  • 8
  • 13
  • Ok, "Response time" is half the answer. It still begs a definition for "Request time". – Berwyn Sep 02 '22 at 22:43
  • Where do you see "request time"? – Solowalker Sep 03 '22 at 23:07
  • See my results "summarised here" link in the question. The very first table is called "Request Statistics". See the Average ms column. Locust outputs two values: "request" and "response". The first table is the request results, the second is the response results. The question is: How are these two related? – Berwyn Sep 05 '22 at 00:26
  • They're the same thing (response times for your requests), showing different statistics. You make a request and then wait for a response. – Solowalker Sep 05 '22 at 14:11
  • Hmmm... How can they be the same thing when their timing result is so different? For example, the top graph shows the average yottalua "Request" time as 1217 ms (average) whereas the bottom graph shows the yottalua "Response time" as 230 ms (50%tile = median). This is from the same test! There is an order of magnitue difference between the two. – Berwyn Sep 06 '22 at 21:53
  • 50%tile is not median. Percentile means that's the value (230ms) where 50% of the values are at or below that number. Average means you add up all the numbers, including ridiculously high and low numbers, and divide them by the number of requests. You'll have to look up examples and definitions online for more information. – Solowalker Sep 07 '22 at 06:36
  • You are completely missing the point. I know what the definitions are. For our purposes they are close enough to identical. But those figures are an order of magnitude different. For median and percentile see https://math.stackexchange.com/questions/2048470/is-50th-percentile-equal-to-median But it's missing the point. – Berwyn Sep 08 '22 at 07:32
  • Sorry, my comment yesterday was incorrect and I didn't notice until after I couldn't edit it anymore. 50%ile is median but it is NOT the average. They are absolutely not close enough to identical, not even close. That's the point. https://data36.com/statistical-averages-mean-median-mode/ https://www.thebalance.com/median-vs-average-what-the-difference-2682237 – Solowalker Sep 08 '22 at 18:29
  • Edited my answer to give specific examples to illustrate the difference between percentiles and average. Hopefully that makes more sense to you. – Solowalker Sep 08 '22 at 19:19
  • You are quite right. The outliers can skew the average. Thanks for persisting. – Berwyn Sep 09 '22 at 21:26