0

This is an interesting situation I have come across. I wrote some code a while back that was synchronous but then I switched to async. The following code resembles the async code.

async def some_coroutine(args, obj): 
   genarg1, genarg2 = args
   for (param1, param2) in some_generator(genarg1, genarg2): 
     await asyncio.sleep(0) # do this to prevent blocking and give control to event loop
     # do work
     for i in enumerate(param2):
        await asyncio.sleep(0) # do this to prevent blocking and give control to event loop 
        obj.somefunc(i, param1)
     pass

I want to refactor the above such that I can make it compatible with some of the non-async code. I used to have it where the for loops can be called in their own functions except this would block the eventloop. I don't want them to take over the event loop without giving control back to the event loop from time to time. I'd like to refactor to something like this but can't figure out how to avoid the blocking aspect of it:

async def some_coroutine(args, obj):
   genarg1, genarg2 = args
   somefunc(genarg1, genarg2, obj)

def somefunc(genarg1, genarg2, obj):
   for (param1, param2) in some_generator(genarg1, genarg2): 
     # do work
     for i in enumerate(param2):
        obj.somefunc(i, param1)
     pass

Clearly, the first code block, the protocol attempted to not block the event loop because the code was in one routine and had await asyncio.sleep(0). But now the refactored code breaks apart the for loop and is blocking and I'm not able to place await asyncio.sleep(0) in somefunc. I'd like to refactor the code this way so I could call it from other functions that don't use eventloops (e.g., test cases) but when an eventloop is used, I'd prefer it to be versatile enough to not block it.

Is this possible or am I just thinking about it wrong (i.e., refactor the code differently)?

LeanMan
  • 474
  • 1
  • 4
  • 18
  • IMHO using `asyncio.sleep(0.0)` to give control back to the event loop momentarily is an indication that something is not quite right, but it's quite difficult to tell without knowing the context because how to use concurrency is very dependent on the context. Also, AFAIK if you want to run non-async code in a mostly async library, you can launch it in a separate thread so that the main event loop is not blocked. See https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor. – Mr. Fegur May 01 '22 at 05:58
  • Interesting. Could you explain in what way something might not be right? Perhaps I need to study that more – LeanMan May 01 '22 at 06:35
  • Looking at the example, that is certainly one way for me to run a synchronous function in async function and perhaps one where I don't need to call `asyncio.sleep(0.0)`. Now I'm curious about the GIL and how that now plays into all of this given that now I am running more than one thread in my code. – LeanMan May 01 '22 at 06:38
  • Python threads are fake threads because only one thread is active at any time because of the GIL. – Mr. Fegur May 01 '22 at 06:45
  • No function in asyncio should block, and must give up the event loop via an await call as it waits for something else to finish. So why do you need additional sleep calls to micromanage when the control is transferred to something else? If you have cpu intensive tasks, offload them to another process or thread, or another service. – Mr. Fegur May 01 '22 at 06:53
  • Interesting thanks! Yea that may make more sense. Additionally, I found this resource regarding the GILs interaction regarding threads and processes with asyncio/eventloop: https://stackoverflow.com/a/70460553/10421103 – LeanMan May 01 '22 at 06:55

0 Answers0