Using Asyncio module to call a url multiple times, getting no performance benefit

Question

I'm trying to use Python's asyncio module (for the first time) to call a URL multiple times. Asyncio is deemed as an alternative to threads so I am thinking of using it. But I'm unable to

a) Write concise code. I have to repeat the same code multiple times[code below]
b) I'm not achieving any performance benefit [time taken is similar to synchronously calling external API]

import requests
import time
import asyncio

start_time = time.time()

async def get_uuid5():
    r = requests.get("https://www.uuidgenerator.net/api/version1")
    print (r.text)

async def get_uuid4():
    await get_uuid5()
    r = requests.get("https://www.uuidgenerator.net/api/version1")
    print (r.text)

async def get_uuid3():
    await get_uuid4()
    r = requests.get("https://www.uuidgenerator.net/api/version1")
    print (r.text)

async def get_uuid2():
    await get_uuid3()
    r = requests.get("https://www.uuidgenerator.net/api/version1")
    print (r.text)


async def get_uuid():
    await get_uuid2()
    r = requests.get("https://www.uuidgenerator.net/api/version1")
    print (r.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(get_uuid())
loop.close()

end_time = time.time()

print("time taken: ", end_time - start_time)

What am I missing? Is my implementation faulty?

Because you `await` another function in most of your `get_uuidN` functions, this seems more sequential than parallel. — jkr, Oct 07 '20 at 18:00
All functions calls are serialized, so they have to wait for each other before return. Unless you call them at the same time, you won't get any benefit from async io. Also `requests.get` should be replaced by async version too. — sardok, Oct 07 '20 at 18:02

Hamza · Accepted Answer · 2020-10-07T18:16:02.197

What am I missing? Is my implementation faulty?

Yes implementation is faulty. You are using awaits wrongly so it actually make them go sequentially. Thats abuse of await and its not intended to use this way. Refer to documentation for more.

Asyncio is deemed as an alternative to threads

No its not. It actually execute each of the loop iteration in a separate executer as soon as it gets to the iteration and moves on without waiting for it to complete (assuming you want to loop through as your implementation suggests) while using threads meaning running every iteration in a separate thread IN PARALLEL to each other.

Your case can be addressed with the help of a background function as follows:

import requests
import time
import asyncio

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)
    return wrapped

@background
def get_uuid(i):
    r = requests.get("https://www.uuidgenerator.net/api/version1")
    print(f"function finished for {i} with output:{r.text}")

start_time = time.time()
for i in range(5):
    get_uuid(i)

end_time = time.time()
print("time taken: ", end_time - start_time)

Produces following output:

time taken:  0.001004934310913086
function finished for 4 with output:fed36736-08c7-11eb-adc1-0242ac120002
function finished for 0 with output:fed4ffec-08c7-11eb-adc1-0242ac120002
function finished for 2 with output:fed8b9f2-08c7-11eb-adc1-0242ac120002
function finished for 1 with output:fed90cfe-08c7-11eb-adc1-0242ac120002
function finished for 3 with output:fed870c8-08c7-11eb-adc1-0242ac120002

This way:

Your code is concise
And has obvious advantage over running it in sequential

Worth noting that the time printed here is not the time taken by loop to complete but merely the time taken to trigger all iterations. Since loop simply triggers executers and once its job is done, this output is produced. Iterations may keep running in background depending on their time of execution. That's why you see iterations' outputs after the loop is already finished.

Credits: This Answer

Thanks, this answer reduces time drastically. So `run_in_executor` is what is helping here? — Coderaemon, Oct 07 '20 at 18:21
Yupp! Coupled with @background signtaure. So it runs in an executor in background — Hamza, Oct 07 '20 at 18:22

score 3 · Answer 2 · answered Oct 07 '20 at 19:02

3

From Python.org: Application developers should typically use the high-level asyncio functions, such as asyncio.run(), and should rarely need to reference the loop object or call its methods. This section is intended mostly for authors of lower-level code, libraries, and frameworks, who need finer control over the event loop behavior.

For your code please use Task. Tasks are used to schedule coroutines concurrently. asyncio.create_task()

answered Oct 07 '20 at 19:02

Aaj Kaal

1,205
1
9
8

Creating a `task` does make code run faster but not by much. It is giving -1 sec improvement. – Coderaemon Oct 10 '20 at 06:31

Using Asyncio module to call a url multiple times, getting no performance benefit

2 Answers2