0

I am working on code that accesses an api that's not made using asyncio along with a discord bot that uses asyncio. In my dsicord commands I created local functions with the blocking parts of the api and want to run them with BaseEventLoop.run_in_executer with a ProcessPoolExecutor object.

Example:

@client.command()
def someCommand(...):
    ... processing where lots of local variables are defined

    def task():
        ... blocking code making use of lots of local variables
    
    loop.run_in_executer(processPool, task)

Some very useful answers are given around the multiprocessing module here. Are there any similar solutions without moving the functions to global space where a lot of unwanted clutter will be created :( ? Having the blocking functions defined locally makes the code more clean and readable. I already have hundreds of functions in my file and don't want to add unnecessary global space clutter.

Hein Gertenbach
  • 318
  • 2
  • 13
  • 1
    How is it more cluttered defining the same function globally? Put a leading `_` on it (to indicate it's not part of the public API), and it won't show up in tab-completion or the like (you could also define `__all__` for the module for the same purpose if you prefer). Also, if the code is blocking, could you just use a `ThreadPoolExecutor`? `ProcessPoolExecutor`s are more important when the code is CPU-bound, but if it's blocking code, it likely isn't, and threads are likely fine (since it's only using local variables, not shared global state). – ShadowRanger Dec 23 '22 at 11:45
  • `I already have hundreds of functions in my file` how about creating more files ? you can search for `python project structure` to see how to structure your project, which is probably the bigger problem, and the error you are seeing is caused by it. – Ahmed AEK Dec 23 '22 at 11:59

1 Answers1

1

your options are to use loky or pathos which can serialize local functions by using cloudpickle or dill (and have a slight overhead over normal python pools), but they are low level APIs unlike concurrent.futures which is a high level API, so you will have to do some wrapping yourself using asyncio.run_in_executor with a threadpool that will be wrapping the low level pools.

Ahmed AEK
  • 8,584
  • 2
  • 7
  • 23
  • *Only* option? `dill` is a thing, and I'm fairly sure there are other options besides that. – ShadowRanger Dec 23 '22 at 11:47
  • @ShadowRanger for a multiprocessing framework you'd have to use `pathos` in order to use `dill` and its behavior is weirder than `loky` ... but i'll add that in the answer. – Ahmed AEK Dec 23 '22 at 11:49
  • Using loky or dill is a solution to make serializing local functions possible, but the solution I opted to which resulted in cleaner code was to make my api library asynchronous – Hein Gertenbach Dec 23 '22 at 14:30