3

I want to run blocking and unblocking tasks together asynchronously. Obviously that it is necessary to use run_in_executor method for blocking tasks from asyncio. Here is my sample code:

import asyncio
import concurrent.futures
import datetime
import time


def blocking():
    print("Enter to blocking()", datetime.datetime.now().time())
    time.sleep(2)
    print("Exited from blocking()", datetime.datetime.now().time())


async def waiter():
    print("Enter to waiter()", datetime.datetime.now().time())
    await asyncio.sleep(3)
    print("Exit from waiter()", datetime.datetime.now().time())


async def asynchronous(loop):
    print("Create tasks", datetime.datetime.now().time())
    task_1 = asyncio.create_task(waiter())

    executor = concurrent.futures.ThreadPoolExecutor(max_workers=3)
    task_2 = loop.run_in_executor(executor, blocking)

    tasks = [task_1, task_2]
    print("Tasks are created", datetime.datetime.now().time())
    await asyncio.wait(tasks)


if __name__ == "__main__":
    try:
        loop = asyncio.get_event_loop()
        loop.run_until_complete(asynchronous(loop))
    except (OSError) as exc:
        sys.exit('Exception: ' + str(exc))

Should I use the same event loop for blocking task in run_in_executor, or it is necessary to use another one? What should I change in my code to make it work asynchronously? Thanks

d.golov
  • 61
  • 1
  • 5
  • In addition to what the answer says, note that you don't need to create a new thread pool for each run. The point of a thread pool is to reuse the threads for efficiency. You can simply pass `None` as the first argument to `run_in_executor` and have it use the thread pool created by asyncio for the event loop. – user4815162342 Sep 02 '20 at 11:45
  • oh, I see. But if I will not use separate threads from custom thread pull, it will block thread with event loop, isn't it? – d.golov Sep 02 '20 at 14:15
  • I'm not saying you should use a thread pool, I'm just saying you don't need to create your own thread pool. Passing `None` to `run_in_executor` will use asyncio's own thread pool which exists for this very purpose. That thread pool still has its own threads and won't block the event pool thread. – user4815162342 Sep 02 '20 at 14:17
  • I see, now. I just was confused with this phrase from official documentation: "The loop.run_in_executor() method can be used with a concurrent.futures.ThreadPoolExecutor to execute blocking code in a different OS thread without blocking the OS thread that the event loop runs in." I thought that by default there is only one thread with event loop, and using your own thread pool you will avoid block of main thread with event loop – d.golov Sep 02 '20 at 14:20
  • You thought correctly, there is indeed only one thread that runs the event loop. This thread pool is a helper thing provided for situations such as yours, to avoid everyone creating their own thread pool and the number of threads growing without bounds. So it's true that you have to use _a_ thread pool, but it doesn't have to be your own - the _thread pool_ provided by asyncio (which is distinct from the thread that runs the event loop) is perfectly fine. – user4815162342 Sep 02 '20 at 14:41
  • @user4815162342, thanks a lot for explanations. It is much more clear for me now. You are great. Thanks – d.golov Sep 02 '20 at 17:30

1 Answers1

4

You must use the same loop. The loop delegates to the executor, which runs tasks is separate threads to the event loop. So you don't have to worry about your blocking tasks blocking the event loop. If you use a separate loop, your async functions from the event loop will not be able to await the results of blocking the functions run in the new loop.

The event loop manages this by creating a future to represent the executor task. It then runs the blocking task in one of the executors threads, and when the executor task returns the result of the future is set and control returned to awaiting function in the event loop (if any).

Dunes
  • 37,291
  • 7
  • 81
  • 97
  • Thanks a lot. So the code above seems to be correct? There one event loop was created and it was used for blocking and non-blocking tasks. – d.golov Sep 02 '20 at 11:18
  • It's good, except that you should avoid passing around the event loop in async functions - it could lead to bugs. Instead, use `asyncio.get_running_loop()` if an async function needs access to the event loop. That's new in 3.7, so use `get_event_loop()` if you using 3.6 or lower. – Dunes Sep 02 '20 at 11:23