0

We have a system that will get ported from C# to Python. My Python experience is limited to making simple tools and the more I read about multi threading, the less I’m confident the switch is a good idea.

The system is dealing with financial data, by blocks of 5 seconds, arriving in real time.

It has one main processing thread, two threads waiting for orders from the main thread and another thread listening to a socket where data is passed to the main thread, affecting its operations on the fly.

The reason for the switch is mostly for unification purpose and not based on tech considerations. In short, politics...

From what I read, Python is not truly multithreaded but can switch pseudo threads on blocking IO, which is not something making any sense in the current context.

The threads could be separated into different processes / apps but then I have to solve the communication problem and since there doesn’t seem to be true threading solution, I can’t have a thread listening to messages while the other one does the work; polling doesn’t really make sense.

So, is my understanding of Python’s abilities correct? Or have things improved compared to what I am reading, which is very messy; I haven’t found a good authority that says “you can do x, you can’t do y” and most of the web post seem so ecstatic about Python that no one seems to be going in depth about things it may not be able to do.

So, to narrow down my question: can I create true threads and use non polling mechanisms to pass messages, like a thread waking up on a socket message, etc in Python? (Doing it in C# is trivial so I’m guessing there has to be some way)

Thomas
  • 10,933
  • 14
  • 65
  • 136
  • "can I create true threads"... no. "and use non polling mechanisms"... [yes](https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#exchanging-objects-between-processes). Using the [`multiprocessing`](https://docs.python.org/3.4/library/multiprocessing.html) module, you can have [one process block until another passes a message, without a `while (not check): sleep` loop.](https://docs.python.org/3.4/library/threading.html#threading.Event.wait) The main thing missing is [shared memory](https://stackoverflow.com/a/16580923/2886575) – Him Jul 20 '19 at 18:09
  • So one python app can have multiple processes that synchronize each other’s? Since there is no shared memory, how do you pass data from one process to another? – Thomas Jul 20 '19 at 18:19
  • All of the `multiprocessing` module's methods essentially perform a memcpy under the hood. If you're on linux, you could use a [ramfs](https://www.jamescoyle.net/how-to/943-create-a-ram-disk-in-linux) if you have a read-only object to share, or a minimal-write object and some extra memory to burn. If you *really* need shared memory, you might write a hunk of code that does threading in C, and create high-level hooks into python, depending on the application. [Stuff using, e.g, `numpy` arrays](https://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.NpyAuxData_CloneFunc) can do this. – Him Jul 20 '19 at 18:36
  • If you just need to pass a bool, then a memcpy is trivial overhead. If you need to share a 1GB matrix, then memcpy sucks, and you have to think outside of the box, and probably do some C coding. – Him Jul 20 '19 at 18:38
  • [multiprocessing also has some basic shared memory stuff](https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#shared-ctypes-objects) for certain object types. [This answer](https://stackoverflow.com/a/10724332/2886575) describes some of the shared-memory capabilities and limitations of the multiprocessing module. – Him Jul 20 '19 at 19:15

0 Answers0