0

From the last post, the duplicate post cannot answer my question.

Right now, I have a function f1() which contains CPU intensive part and async IO intensive part. Therefore f1() itself is an async function. How can I run the whole f1() with given timeout? I found the method provided in the post cannot solve my situation. For the following part, it shows RuntimeWarning: coroutine 'f1' was never awaited handle = None # Needed to break cycles when an exception occurs.

import asyncio
import time
import concurrent.futures

executor = concurrent.futures.ThreadPoolExecutor(1)


async def f1():
    print("start sleep")
    time.sleep(3)  # simulate CPU intensive part
    print("end sleep")

    print("start asyncio.sleep")
    await asyncio.sleep(3)  # simulate IO intensive part
    print("end asyncio.sleep")


async def process():
    print("enter process")
    loop = asyncio.get_running_loop()
    await loop.run_in_executor(executor, f1)


async def main():
    print("-----f1-----")
    t1 = time.time()
    try:
        await asyncio.wait_for(process(), timeout=2)
    except:
        pass
    t2 = time.time()
    print(f"f1 cost {(t2 - t1)} s")


if __name__ == '__main__':
    asyncio.run(main())

From previous post, loop.run_in_executor can only work for normal function not async function.

maplemaple
  • 1,297
  • 6
  • 24
  • You need to run the CPU-intensive part with `run_in_execturo`, not the whole coroutine. Furthermore, if the code is CPU-intensive, you need to run it inside a `ProcessPoolExecutor`. By default (i.e., when you pass `None`) it uses a `ThreadPoolExecutor` (or in your case where you explicitly create one). – dirn Dec 02 '22 at 14:04

2 Answers2

0

one way to do it is to make process not an async function, so it can run in another thread, and have it start an asyncio loop in the other thread to run f1.

note that starting another loops means you cannot share coroutines and futures between the two loops.

import asyncio
import time
import concurrent.futures

executor = concurrent.futures.ThreadPoolExecutor(1)


async def f1():
    print("start sleep")
    time.sleep(3)  # simulate CPU intensive part
    print("end sleep")

    print("start asyncio.sleep")
    await asyncio.sleep(3)  # simulate IO intensive part
    print("end asyncio.sleep")


def process():
    print("enter process")
    asyncio.run(asyncio.wait_for(f1(),2))

async def main():
    print("-----f1-----")
    t1 = time.time()
    try:
        loop = asyncio.get_running_loop()
        await loop.run_in_executor(executor, process)
    except:
        pass
    t2 = time.time()
    print(f"f1 cost {(t2 - t1)} s")


if __name__ == '__main__':
    asyncio.run(main())

-----f1-----
enter process
start sleep
end sleep
start asyncio.sleep
f1 cost 3.0047199726104736 s

keep in mind that you must wait for any IO to return f1 to the eventloop so the future can be cancelled, you cannot cancel the CPU-intensive part of the code unless it does something like await asyncio.sleep(0) which returns to the event-loop momentarily, which is why time.sleep cannot be cancelled.

Ahmed AEK
  • 8,584
  • 2
  • 7
  • 23
  • In your code, the f1 cost `3s`. However, I want to stop after `2s`. – maplemaple Dec 02 '22 at 08:50
  • @maplemaple you can't .... python itself isn't running during that time, and the OS isn't servicing the thread on any core. https://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread – Ahmed AEK Dec 02 '22 at 08:51
  • @maplemaple you should probably either implement a kill switch that is periodically checked ... similar to `await asyncio.sleep(0)`, or use multiprocessing instead of threading. – Ahmed AEK Dec 02 '22 at 09:02
0

I have explained the cause of the issue. You should remove or replace the time.sleep at f1 as it blocks the thread, and asyncio.wait_for cannot handle the timeout.

Regarding to the RuntimeWarning

RuntimeWarning: coroutine 'f1' was never awaited handle = None # Needed to break cycles when an exception occurs.

It occurs because the loop.run_in_executor expects a non-async function as a second argument.

Artyom Vancyan
  • 5,029
  • 3
  • 12
  • 34