7

I have created a dictionary in my Python application where I save the data and I have two tasks that run concurrently and get data from external APIs. Once they get the data, they update the dictionary - each with a different key in the dictionary.

I want to understand if the dictionary is async safe or do I need to put a lock when the dictionary is read/updated?

The tasks also read the last saved value each time.

my_data = {}
asyncio.create_task(call_func_one_coroutine)
asyncio.create_task(call_func_two_coroutine)

async def call_func_one_coroutine():
  data =  await goto_api_get_data()
  my_data['one'] = data + my_data['one']


async def call_func_two_coroutine():
  data =  await goto_api_another_get_data()
  my_data['two'] = data + my_data['two']
InfoLearner
  • 14,952
  • 20
  • 76
  • 124

1 Answers1

21

I want to understand if the dictionary is async safe or do I need to put a lock when the dictionary is read/updated?

Asyncio is based on cooperative multitasking, and can only switch tasks at an explicit await expression or at the async with and async for statements. Since update of a single dictionary can never involve awaiting (await must completes before the update begins), it is effectively atomic as far as async multitasking is concerned, and you don't need to lock it. This applies to all data structures accessed from async code.

To take another example where there is no problem:

# correct - there are no awaits between two accesses to the dict d
key = await key_queue.get()
if key in d:
    d[key] = calc_value(key)

An example where a dict modification would not be async-safe would involve multiple accesses to the dict separated by awaits. For example:

# incorrect, d[key] could appear while we're reading the value,
# in which case we'd clobber the existing key
if key not in d:
    d[key] = await read_value()

To correct it, you can either add another check after the await, or use an explicit lock:

# correct (1), using double check
if key not in d:
    value = await read_value()
    # Check again whether the key is vacant. Since there are no awaits
    # between this check and the update, the operation is atomic.
    if key not in d:
        d[key] = value

# correct (2), using a shared asyncio.Lock:
async with d_lock:
    # Single check is sufficient because the lock ensures that
    # no one can modify the dict while we're reading the value.
    if key not in d:
        d[key] = await read_value()
user4815162342
  • 141,790
  • 18
  • 296
  • 355
  • Thank you. That makes sense. Assume there are 2 funcs. when we do await func1() then what are the sequence of steps that event loop performs? 1. Execute func1, run until it sees await or completes. If it sees await then pause func1 and switch and run func2 or 2. Pause func1, switch and run func2, once func2 does await or completes then resume func1? Is there a way to see the code for await? – InfoLearner Nov 27 '20 at 22:10
  • @InfoLearner That's a separate question, but if you want to understand how await works, take a look at [this lecture](https://www.youtube.com/watch?v=MCs5OvhV9S4) (which uses `yield from`, but the exact same applies to `await`) or [this lecture](https://www.youtube.com/watch?v=7JtNiwCH_OA), which is specifically about desugaring `await`. – user4815162342 Nov 28 '20 at 10:40
  • 1
    @InfoLearner If I understand your question correctly (you didn't specify how func2 is actually started), the first variant is what happens. When you `await func1()`, Python immediately starts executing func1 without dropping to the event loop. It is only if `func1` chooses to suspend that the awaiter will suspend too, and so will the awaiter's awaiter, and so on, ultimately reaching the event loop. The event loop will then decide which coroutine to run next, if any. (If no coroutines are runnable, the event loop will poll registered file descriptors and sleep until the next IO/timeout event.) – user4815162342 Nov 28 '20 at 18:19