How to pass messages/information between process using concurrent.futures.ProcessPoolExecutor

Question

I am trying to pass messages between my processes. I have a process that is a client to a server. I want other processes to pass information to the client to pass it to the server. Think of the server as a data logger. I am using concurrent.futures.ProcessPoolExecutor. The processes are working continuously in a while True.

Any hint in how to connect the processes is appreciated,

Thank you.

Edit: I found the answer. All credit goes to whomever posted the following post here.

import multiprocessing
qm = multiprocessing.Manager()
pqueue = qm.Queue()

Answer 1

Processes in general are not able to talk to each other through python's standard library. The memory space for the process is allocated by the OS and in general does not overlap. Parent processes (like your original python process) have some limited visibility of child processes ( like the processes created using ProcessPoolExecutor), but not the ability to pass messages between them.

You have a couple options here.

1) Use threads. If you don't actually need parallelism (ex using all cores of a multi-core system), this option might work. Threads all run within a single process and have shared memory space. So you can write to an object from one thread and read from it in another. concurrent.futures.ThreadPoolExecutor will allow you to do this.

2)Use the underlying OS. In general, when two processes running on the same system need to communicate, they do it over ports. The process that is receiving messages will bind to a port. The process that is sending messages will then establish a socket connection over that port, and use this connection to communicate. Refer to https://docs.python.org/3.7/howto/sockets.html for more details.

3)Use a message broker like rabbitMQ or reddis. These are external processes that facilitate communication between processes.