12

We are using Locust to for load testing rest api services behind elastic load balancing. I came across this article regarding load balancing and auto scaling, which is something we are testing.

Locust is using python-requests which is using urllib3, so my question is if python-requests does a dns query for every connect, and if not, is it configurable?

djonsson
  • 603
  • 5
  • 14
  • Urllib3 is probably using `socket.getaddrinfo` which should be using the `getaddrinfo` of the OS you are using according to this [answer](http://stackoverflow.com/a/16621318/6061947) on another SO question. So it should cache the results depending on the OS for each subsequent request to the same hostname. – Cory Shay Mar 18 '16 at 16:41
  • And according to this question: http://stackoverflow.com/questions/11020027/dns-caching-in-linux caching is disabled on most Linux configurations. – djonsson Mar 19 '16 at 11:50

3 Answers3

5

Locust is using python requests that is using urllib3 that is using socket.getaddrinfo which has DNS caching disabled according to this SO thread (given that your test machine runs linux).

Community
  • 1
  • 1
djonsson
  • 603
  • 5
  • 14
1

python-requests does a dns query for every connect.

To disable this you can use a dns cache.

Now you can enable systemd-resolved with systemctl enable systemd-resolved

more info - https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

Tal Folkman
  • 2,368
  • 1
  • 7
  • 21
1

Yes, the Python requests lib perform a DNS query at every request;

However, you can improve that behavior using requests-cache.

requests-cache is a persistent HTTP cache that provides an easy way to get better performance with the python requests library.

It's really easy to use, 1 minute, 2 lines of code and you're ready to go.

import requests
import requests_cache

requests_cache.install_cache('my_simple_cache')

Your subsequent python requests calls should automagically use the cache now. And there are more fine-grained options available if you want, like customizing expire time, etc.

(This solved a problem I was having trying to batch process something that suddenly stopped working after 10K calls, not because the service was unavailable, but because the DNS requests would get refused at my intranet DNS server.)

Vinícius M
  • 554
  • 6
  • 15
  • After various attempts with systemd-resolved, this helps reduce the DNS calls substantially. ```requests_cache.install_cache(backend='memory', expire_after=600) ``` – sp1111 Jul 05 '23 at 00:05
  • AFAICT, requests-cache is a cache for the payload, not a DNS cache. – MattK Aug 18 '23 at 18:38
  • @MattK Well, then now you know more! It does provide hostname-to-ip dns caching as well. Like I described in my answer, and like sp1111 mentioned in his comment. – Vinícius M Aug 22 '23 at 12:57