4

Is it possible to use Python3 asyncio package with Boost.Python library?

I have CPython C++ extension that builds with Boost.Python. And functions that are written in C++ can work really long time. I want to use asyncio to call these functions but res = await cpp_function() code doesn't work.

  • What happens when cpp_function is called inside coroutine?
  • How not get blocked by calling C++ function that works very long time?

NOTE: C++ doesn't do some I/O operations, just calculations.

Illia Ananich
  • 353
  • 4
  • 16

1 Answers1

4

What happens when cpp_function is called inside coroutine?

If you call long-running Python/C function inside any of your coroutines, it freezes your event loop (freezes all coroutines everywhere).

You should avoid this situation.

How not get blocked by calling C++ function that works very long time

You should use run_in_executor to run you function in separate thread or process. run_in_executor returns coroutine that you can await.

You'll probably need ProcessPoolExecutor because of GIL (I'm not sure if ThreadPoolExecutor is option in your situation, but I advice you to check it).

Here's example of awaiting long-running code:

import asyncio
from concurrent.futures import ProcessPoolExecutor
import time


def blocking_function():
    # Function with long-running C/Python code.
    time.sleep(3)
    return True


async def main():        
    # Await of executing in other process,
    # it doesn't block your event loop:
    loop = asyncio.get_event_loop()
    res = await loop.run_in_executor(executor, blocking_function)


if __name__ == '__main__':
    executor = ProcessPoolExecutor(max_workers=1)  # Prepare your executor somewhere.

    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()
Mikhail Gerasimov
  • 36,989
  • 16
  • 116
  • 159
  • Thank you for your answer :) Can you help understand why have you defined `max_workers` to `1`, and why you have used `ProcessPoolThreads`? As Python documentation says treads are used for I/O operations, while processes for CPU work. And in my case coroutine calls `cpp_function` just to get one string from another string. Can this `cpp_function` just take all CPU resources and then any asyncio doesn't make sense? What pros can asyncio (and Executors) have in my case? – Illia Ananich Jun 12 '17 at 18:49
  • @ІлляАнаніч You're welcome! 1) You can change `max_workers` if you're planning to use this executor for multiple tasks. 2) I don't know what exactly `cpp_function` doing, but if it somehow process string - it is CPU operation. I/O operations are usually something network-related: reading from socket, for example. – Mikhail Gerasimov Jun 12 '17 at 19:14
  • 1
    3) That's why I think you should use `ProcessPoolExecutor`. But it's easy to check if I'm wrong: try to use `ThreadPoolExecutor` for `cpp_function` and run some simple task parallely (it can just print something every second). If this task would be freezed at time when `cpp_function` is executing, then threads are not an option. – Mikhail Gerasimov Jun 12 '17 at 19:14
  • Hmm, I have tried your solution, but got this error: `TypeError: can't pickle Boost.Python.function objects` – Illia Ananich Jun 13 '17 at 17:58
  • @ІлляАнаніч, what happens: Python get some data from Boost function in one process and tries to pass it to main process. But not every kind of data can be passed from process to process. Data should be pickable, see this [link](https://stackoverflow.com/a/32088533/1113207) for more info. To make things work you shouldn't run Boost function in executor direcly, instead you should probably run some sort of "proxy" function that would convert result of boost function to some pickable object and return that object as result. I don't know if there's some other way to pass data between processes. – Mikhail Gerasimov Jun 13 '17 at 20:50