Combine awaitables like Promise.all

Question

In asynchronous JavaScript, it is easy to run tasks in parallel and wait for all of them to complete using Promise.all:

async function bar(i) {
  console.log('started', i);
  await delay(1000);
  console.log('finished', i);
}

async function foo() {
    await Promise.all([bar(1), bar(2)]);
}

// This works too:
async function my_all(promises) {
    for (let p of promises) await p;
}

async function foo() {
    await my_all([bar(1), bar(2), bar(3)]);
}

I tried to rewrite the latter in python:

import asyncio

async def bar(i):
  print('started', i)
  await asyncio.sleep(1)
  print('finished', i)

async def aio_all(seq):
  for f in seq:
    await f

async def main():
  await aio_all([bar(i) for i in range(10)])

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

But it executes my tasks sequentially.

What is the simplest way to await multiple awaitables? Why doesn't my approach work?

score 108 · Accepted Answer · edited Nov 06 '21 at 05:49

108

The equivalent would be using asyncio.gather:

import asyncio

async def bar(i):
  print('started', i)
  await asyncio.sleep(1)
  print('finished', i)

async def main():
  await asyncio.gather(*[bar(i) for i in range(10)])

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

Why doesn't my approach work?

Because when you await each item in seq, you block that coroutine. So in essence, you have synchronous code masquerading as async. If you really wanted to, you could implement your own version of asyncio.gather using loop.create_task or asyncio.ensure_future.

EDIT

The original answer used the lower-level asyncio.wait.

edited Nov 06 '21 at 05:49

charmoniumQ

5,214
5
33
51

answered Dec 20 '15 at 02:17

Jashandeep Sohi

4,903
2
23
25

21

please mention asyncio.gather too – Andrew Svetlov Dec 20 '15 at 11:16
Is the main difference between `wait` and `gather` that `wait` allows a `timeout` argument? – Thomas Ahle Feb 27 '18 at 12:27
asyncio.wait is the more low level way of waiting. Also in 3.8 passing coroutines to wait directly is deprecated. Please rewrite your answer or update with a snippet using asyncio.gather as it's closer to Promise.all. – Rémi Benoit Jul 07 '21 at 14:30
What's the difference if the function returns a value? – Jorge Tovar May 19 '22 at 22:07
Very good. I got `Task got bad yield: ` because I was using `gather(f(x) for x in xs)` instead of `gather(*[f(x) for x in xs])` – Gavin Haynes Aug 31 '22 at 21:59

score 23 · Answer 2 · answered Apr 19 '19 at 02:12

I noticed that asyncio.gather() may be a better way to await other than asyncio.wait() if we want ordered results.

As the docs indicates, the order of result values from asyncio.gather() method corresponds to the order of awaitables in aws. However, the order of result values from asyncio.wait() won't do the same thing.You can test it.

James T. · Answer 3 · 2022-03-07T21:46:57.203

https://docs.python.org/3/library/asyncio-task.html#asyncio.gather

asyncio.gather() will return the list of output from each async function calls.

import asyncio

async def bar(i):
    print('started', i)
    await asyncio.sleep(1)
    print('finished', i)
    return i

async def main():
    values = await asyncio.gather(*[bar(i) for i in range(10)])
    print(values)

asyncio.run(main())

This method, gather, takes arbitrary number of args for the concurrent jobs instead of a list, so we unpack.

It's very common to need this intermediate value, values in my eg, instead of designing your function/method to have side effects.

score 3 · Answer 4 · answered Nov 19 '22 at 21:03

3

As of Python 3.11:

A more modern way to create and run tasks concurrently and wait for their completion is asyncio.TaskGroup.

... although, I couldn't find any reason why TaskGroup should be preferred over gather

answered Nov 19 '22 at 21:03

kym

818
7
12

Hans Bouwmeester · Answer 5 · 2023-07-29T02:32:54.193

To make this work the only change needed is to use asyncio.as_completed() to create an iterator of coroutines (awaitables) you can use for the for-loop:

async def asyncio_all(aws):
    for future in asyncio.as_completed(aws)
        await future

When applied to your code:

import asyncio

async def bar(i):
  print('started', i)
  await asyncio.sleep(1)
  print('finished', i)

async def aio_all(seq):
  for f in asyncio.as_completed(seq):
    await f

async def main():
  await aio_all([bar(i) for i in range(10)])

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

started 1
started 0
started 2
started 6
started 8
started 7
started 5
started 9
started 4
started 3
finished 1
finished 0
finished 2
finished 6
finished 8
finished 7
finished 5
finished 9
finished 4
finished 3

This comes closest to JavaScript Promise.all. However, using asyncio.gather or asyncio.TaskGroup (from v3.11) seems to be the more Pythonic way of doing it.

Combine awaitables like Promise.all

5 Answers5

Linked