How is Python multithreading efficient with the non-async requests library?

Question

I would just like to understand, I'm using a Python script that sends 3 HTTP GET requests to an URL that I made to wait for 5 seconds.

I am using the requests library and the threading Python module like so:

#!/usr/bin/env python3

import time
import requests
from threading import Thread


url = 'https://mydebug-url.example.com/sleep/5'
payload = {}

def myfunc(i):
    response = requests.get(url, json=payload)
    print(f'Got back from {i} {response.status_code}. Response was {response.json()}')


if __name__ == '__main__':

    start = time.time()

    threads = []
    for i in [1, 2, 3]:
        thread = Thread(target=myfunc, args=[i])
        threads.append(thread)

    for thread in threads:
        thread.start()

    # Wait for all threads to finish
    for thread in threads:
        thread.join()

    end = time.time()

    print(f'took {end-start} seconds')

And the output is:

Got back from 2 200. Response was {'message': 'Slept 5 seconds'}
Got back from 1 200. Response was {'message': 'Slept 5 seconds'}
Got back from 3 200. Response was {'message': 'Slept 5 seconds'}
took 5.35151219367981 seconds

Now my question is, how is that efficient? So this code in the sample, using threading, takes a little more than 5 seconds so it's almost like doing them in parallel.

What puzzles me is, that it's kind of efficient, although:

Python has GIL so it would have executed them in a single thread
requests library is blocking (non async) so async wouldn't have worked either to benefit from one thread jumping rapidly between the 3 tasks while waiting.

So, what am I missing? where I can read more about it?

My interest is, I could use more of threading, still get the performance benefits, instead of changing to a async library to do requests, in case that is proven more difficult in the current codebase.

Thanks!

While waiting e. g. for data from network, the GIL is released. — Michael Butscher, Aug 31 '23 at 16:35
"Python has GIL so it would have executed them in a single thread" that isn't what the GIL does. The GIL forces **only one thread to hold the python interpreter**, so Python bytecode is never executed in parallel. But multiple threads are still allowed. — juanpa.arrivillaga, Aug 31 '23 at 16:37
A different feature of async is useful. Python's executor releases and reacquires the GIL regularly to give other python threads a chance to run. That means that resources shared among threads may need to be locked or may update suprisingly. With pure async, you are guaranteed that other code doesn't run between your await calls, reducing the need for locks. Using locks is error prone because we tend to ignore edge and race conditions. — tdelaney, Aug 31 '23 at 16:45

How is Python multithreading efficient with the non-async requests library?

0 Answers0