asyncio methods not executing asynchronously

Question

I have a following code:

def a:
    parts = asyncio.run(b())
    print(parts)

async def b():
    res = await asyncio.gather(*c(i) for i in range(4))

async def c(i):
    for j in range(5):
        print(j, end=' ')
    return i

I want c() to execute four times at the same time. Yet timing the funcions, as well as the logs, suggest that they are exeucting sequentially:

0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4

[0, 1, 2, 3]

How can I change the code so that the c() method is running 4 times in parallel?

Does this answer your question? [Does await always give other tasks a chance to execute?](https://stackoverflow.com/questions/59996493/does-await-always-give-other-tasks-a-chance-to-execute) — MisterMiyagi, Feb 17 '21 at 15:10
I don't know, it's too complicated. Basically, HTF's answer shows how to change the code so that the methods *seem* to be executed in parallel (as in, the output will be `0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4`). But the problem I have is that the execution still takes the same amount of time. I have many cores/processors, I want the code to run 4 times faster. — momo, Feb 17 '21 at 15:28
``async`` is single-core by design. It *interleaves* processing, it does not *parallelise* processing. ``async`` will only speed up I/O bound tasks, not CPU bound tasks. — MisterMiyagi, Feb 17 '21 at 15:30
Note that your code isn't doing anything really. It mostly writes to ``stdout``, which is synchronised (i.e. non-parallel) by design. It is unsuitable to test speedup from concurrency/parallelism. — MisterMiyagi, Feb 17 '21 at 15:31
my code is just an example simplified to only leave the parts relevant to asyncio. actual code is way more complex and does a lot of stuff that could be parallelized. so you're saying that asyncio is not the way to go, because it doesn't allow parallel execution? what could I use instead? — momo, Feb 17 '21 at 15:36

HTF · Accepted Answer · 2021-02-17T15:36:57.823

0

Adding just an async keyword in front of a function doesn't make it asynchronous. You can try something like this:

test.py:

import asyncio

def a():
    parts = asyncio.run(b())
    print(parts)

async def b():
    return await asyncio.gather(*(c(i) for i in range(4)))

async def c(i):
    for j in range(5):
        await asyncio.sleep(0)
        print(j, end=' ')
    return i

a()

Test:

python test.py
0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 [0, 1, 2, 3]

By adding asyncio.sleep(0) forces the async function to yield control to the event loop and allows the other coroutines to run.

Update: Wed 17 Feb 15:25:50 UTC 2021

Python docs states the following:

An event loop runs in a thread (typically the main thread) and executes all callbacks and Tasks in its thread. While a Task is running in the event loop, no other Tasks can run in the same thread. When a Task executes an await expression, the running Task gets suspended, and the event loop executes the next Task.

So all tasks are running in a single thread and asyncio.sleep() always suspends the current task, allowing other tasks to run.

Please note that concurrency and parallelism are similar terms, but they are not the same thing if you need to run multiple tasks at the same time across multiple CPU cores use multiprocessing.

edited Feb 17 '21 at 15:36

answered Feb 17 '21 at 15:01

HTF

6,632
6
30
49

I don't get it. Why does adding an instruction that does nothing (sleep for 0 seconds) change it? Is this some hack? Is that expected behaviour? Way more importantly - while it does change the *order* of prints, the *timing* remains unchanged. I have 4 cores, I want it to run 4 times faster, how can I do that? – momo Feb 17 '21 at 15:17
@momo please see my update: `Wed 17 Feb 15:25:50 UTC 2021`. – HTF Feb 17 '21 at 15:37
thank you! I'll check multiprocessing.Pool() then – momo Feb 17 '21 at 15:43
@momo You probably want to use [`run_in_executor`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor) and pass it an instance of [`ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor) as the first argument. That executor uses `multiprocessing.Pool` under the hood, but is (when used through `run_in_executor`) compatible with asyncio. – user4815162342 Feb 17 '21 at 16:57

asyncio methods not executing asynchronously

1 Answers1