Scheduled tasks in ensure_future are overlapping by the last one

Question

I need a new handler that use another existing handler for processing multiple requests at once:

async def process_multiple_queries(request):
    init_request = request.clone()
    body = await request.json()

    for query in body["queries"]:

        @asyncio.coroutine
        def fake_json(self):
            return query

        # Replace request body on new fake json
        R = init_request.clone()
        R.json = fake_json.__get__(R, Request)

        # Process request in the background task
        asyncio.ensure_future(process_query(R))

async def process_query(request):
    body = await request.json()
    # ... Process body ...
    return web.json_response({"result": "OK"})

However, results of all scheduled tasks are equal to the result of the last task, i.e. bodies of all requests passed in process_query are equal only to the body of the last request.

But if I add asyncio.sleep(5) after asyncio.ensure_future(process_query(R)), all tasks are processed properly and produce different results.

How to fix it?

user4815162342 · Answer 1 · 2018-10-04T06:26:49.240

This is the well-known Python gotcha where a closure captures variables by name and not by value. As a result, all instances of the fake_json coroutine refer to the same query variable, and so all end up referencing its last known value.

To fix the issue, you need to change the definition:

        @asyncio.coroutine
        def fake_json(self):
            return query

to a formulation that captures the value of query, such as:

        @asyncio.coroutine
        def fake_json(self, query=query):
            return query

await asyncio.sleep(5) helps because it allows the process_query to run to completion before the value of query changes from under its feet. You'd get the same effect with await process_query(R).

Two points unrelated to the question:

The @asyncio.coroutine decorator is deprecated and there is no reason to use it in new code (unless you specifically aim for compatibility with pre-3.5 Python). Instead, simply define fake_json using async def.
You don't need to call __get__ to bind a self to fake_json, which isn't even using it. You can define a fake_json that doesn't self and simply assign R.json = fake_json.

Scheduled tasks in ensure_future are overlapping by the last one

1 Answers1