Using contextvar to keep track of async loop in Python

Question

I am trying to work with a simple async example in Python, largely following this excellent answer here.

My goal is to set up a context variable and keep track of the series of calls by continuously appending to it. I know that context variables can be accessed with the .get() method and their values altered with the .set() method. In the below case however, the variable doesn't get modified despite the series of calls to the function sum() that is apparent from the console.

Edit: Based on Michael Butscher's comment below I replaced the original context variable (which was a string) with a list: output_list and modified the list iteratively using .append(). This now does enable me to view the final output but not the intermediate ones in the individual sum() methods.

Full code:

import asyncio
import contextvars
import time

output_list = contextvars.ContextVar('output_list', default=list())

async def sleep():
    print(f'Time: {time.time() - start:.2f}')
    await asyncio.sleep(1)

async def sum(name, numbers):
    total = 0
    for number in numbers:
        print(f'Task {name}: Computing {total}+{number}')
        await sleep()
        total += number
    output_list.set(output_list.get().append(f"{name}"))
    print(f'Task {name}: Sum = {total}\n')
    print(f'Partial output from task {name}:', output_list.get())

start = time.time()

loop = asyncio.get_event_loop()
tasks = [
    loop.create_task(sum("A", [1, 2])),
    loop.create_task(sum("B", [1, 2, 3])),
]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

end = time.time()
print(f'Time: {end-start:.2f} sec')
print("Final output_str =", output_list.get())

How can I iteratively follow the expansion of the context variable list output_list?

My desired console output is:

Task A: Computing 0+1
Time: 0.00
Task B: Computing 0+1
Time: 0.00
Task A: Computing 1+2
Time: 1.02
Task B: Computing 1+2
Time: 1.02
Task A: Sum = 3

Partial output from task A: ['A']
Task B: Computing 3+3
Time: 2.02
Task B: Sum = 6

Partial output from task B: ['A', 'B']
Time: 3.03 sec
Final output_str = ['A', 'B']

Instead, I am getting:

Task A: Computing 0+1
Time: 0.00
Task B: Computing 0+1
Time: 0.00
Task A: Computing 1+2
Time: 1.02
Task B: Computing 1+2
Time: 1.02
Task A: Sum = 3

Partial output from task A: None
Task B: Computing 3+3
Time: 2.02
Task B: Sum = 6

Partial output from task B: None
Time: 3.03 sec
Final output_str = ['A', 'B']

Each task has its own context. Instead of a string you can set "output_str" to a list before creating the tasks and then only modify the list instead of replacing it. — Michael Butscher, Jan 18 '22 at 02:36
@MichaelButscher thank you, that's very useful advice. I edited my comment. I manage now to get the final output in order but I find that I still cannot record the intermediate states. — lazarea, Jan 18 '22 at 08:22
During the tasks you must not modify the content of the context variable (using "set" method), only modify the list. By the way: (1) The result of "list.append" is always "None". (2) Instead of setting a list as default for the context variable set the list explicitly before creating the tasks. Otherwise all future tasks will use the same list which may not be desired. — Michael Butscher, Jan 18 '22 at 09:30
Thank you @MichaelButscher. So does it mean there is actually no need to define `contextvars` at all? — lazarea, Jan 18 '22 at 12:25
If you have to manage multiple contexts with different variable contents (e. g. different lists) you need context variables. In your code a global variable would work as well. — Michael Butscher, Jan 18 '22 at 13:14

Edher Carbajal · Accepted Answer · 2022-01-19T00:27:32.010

According to the asyncio documentation:

Tasks support the contextvars module. When a Task is created it copies the current context and later runs its coroutine in the copied context.

So, if you declare at the top of your program cvar = contextvars.ContextVar('cvar', default='x'), when you create a task, this will copy the current context, and if you modify cvar whithin it will just affect the copy but no the original context. That's the main reason why you got ''(empty string) at your final output.

To achieve the "track" you want you must use a global variable in order to modify it anywhere. But if you want to play around with asyncio and contextvars to see how it works, see the example below:

import asyncio
import contextvars
import time

output = contextvars.ContextVar('output', default='No changes at all') 

async def sleep():
    print(f'Time: {time.time() - start:.2f}')
    await asyncio.sleep(1)

async def sum(name, numbers):
    total = 0
    for number in numbers:
        print(f'Task {name}: Computing {total}+{number}')
        await sleep()
        total += number
        output.set(output.get()+name) #Here we modify the respective context
    print(f'Task {name}: Sum = {total}\n')
    print(f'Partial output from task {name}:', output.get())
    return output.get() #Here we return the variable modified
start = time.time()

# main() will have its own copy of the context
async def main():
    output.set('Changed - ') # Change output var in this function context
    # task1 and task2 will copy this context (In this contect output=='Changed - ')
    task1 = asyncio.create_task(sum("A", [1, 2])) #This task has its own copy of the context of main()
    task2 = asyncio.create_task(sum("B", [1, 2, 3])) #This task also has its own copy of the context of main()
    done, pending = await asyncio.wait({task1,task2})
    resultTask1 = task1.result() # get the value of return of task1
    resultTask2 = task2.result() # get the value of return of task1
    print('Result1: ', resultTask1)
    print('Result2: ', resultTask2)
    print('Variable output in main(): ',output.get()) # However, output in main() is sitill 'Changed - '
    output.set(output.get()+'/'+resultTask1+'/'+resultTask2) #Modify the var in this context
    print('Variable modified in main(): ', output.get())
    return output.get() #Return modified value

x = asyncio.run(main()) # Assign the return value to x

end = time.time()
print(f'Time: {end-start:.2f} sec')
print("Final output (without changes) =", output.get())
output.set(x)
print("Final output (changed) =", output.get())

##### OUTPUT #####
# Time: 0.00
# Task B: Computing 0+1
# Time: 0.00
# Task A: Computing 1+2
# Time: 1.01
# Task B: Computing 1+2
# Time: 1.01
# Task A: Sum = 3

# Partial output from task A: Changed - AA
# Task B: Computing 3+3
# Time: 2.02
# Task B: Sum = 6

# Partial output from task B: Changed - BBB
# Result1:  Changed - AA
# Result2:  Changed - BBB
# Variable output in main():  Changed -
# Variable modified in main():  Changed - /Changed - AA/Changed - BBB
# Time: 3.03 sec
# Final output (without changes) = No changes at all
# Final output (changed) = Changed - /Changed - AA/Changed - BBB

As you can see, it is impossible to modify the same variable at the same time. While task1 is modifying its copy, task2 is modifying its copy too.

Using contextvar to keep track of async loop in Python

1 Answers1