1

What is the best way to exchange huge amount of data between processes in python. I am working on the image processing algorithms and I would like to send image frames between processes.

Leopoldo
  • 795
  • 2
  • 8
  • 23
  • I had a similar question a while back: http://stackoverflow.com/questions/20971191/sharing-state-between-forked-worker-processes-in-a-high-performance-environment. Unfortunately I didn't find a satisfactory solution. I ended up running all code in a single process. That wasn't an issue because I was able to run multiple experiments, with different parameter settings, at the same time. – mbatchkarov Dec 22 '14 at 13:25
  • 2
    Use `shm_open` maybe? It would seem more efficient to use shared memory rather than moving vast quantities of data around needlessly as messages. – Mark Setchell Dec 22 '14 at 14:10

1 Answers1

1

For interprocess communication between Python processes you should use the asynchronous message queuing communication protocol.

Python has great support for a variety of systems implementing this as ZeroMQ and RabbitMQ.

Or you can check out the listener and client objects of the multiprocessing library.

Riccardo
  • 1,490
  • 2
  • 12
  • 22