Will I run into trouble with python's Global Interpreter Lock?

Question

I am aware that this question is rather high-level and may be vague. Please ask if you need any more details and I will try to edit.

I am using QuickFix with Python bindings to consume high-throughput market data from circa 30 markets simultaneously. Most of computing the work is done in separate CPUs via the multiprocessing module. These parallel processes are spawned by the main process on startup. If I wish to interact with the market in any way via QuickFix, I have to do this within the main process, thus any commands (to enter orders, for example) which come from the child processes must be piped (via an mp.Queue object we will call Q) to the main process before execution.

This raises the problem of monitoring Q, which must be done within the main process. I cannot use Q.get(), since this method blocks and my entire main process will hang until something shows up in Q. In order to decrease latency, I must check Q frequently, on the order of 50 times per second. I have been using the apscheduler to do this, but I keep getting Warning errors stating that the runtime was missed. These errors are a serious issue because they prevent me from easily viewing important information.

I have therefore refactored my application to use the code posted by MestreLion as an answer to this question. This is working for me because it starts a new thread from the main process, and it does not print error messages. However, I am worried that this will cause nasty problems down the road.

I am aware of the Global Interpreter Lock in python (this is why I used the multiprocessing module to begin with), but I don't really understand it. Owing to the high-frequency nature of my application, I do not know if the Q monitoring thread and the main process consuming lots of incoming messages will compete for resources and slow each other down.

My questions:

1. Am I likely to run into trouble in this scenario?

2. If not, can I add more monitoring threads using the present approach and still be okay? There are at least two other things I would like to monitor at high frequency.

Thanks.

Answer 1

@MestreLion's solution that you've linked creates 50 threads per second in your case.

All you need is a single thread to consume the queue without blocking the rest of the main process:

import threading

def consume(queue, sentinel=None):
    for item in iter(queue.get, sentinel):
        pass_to_quickfix(item)

threading.Thread(target=consume, args=[queue], daemon=True).start()

GIL may or may not matter for performance in this case. Measure it.

Answer 2

Without knowing your scenario, it's difficult to say anything specific. Your question suggests, that the threads are waiting most of the time via get, so GIL is not a problem. Interprocess communication may result in problems much earlier. There you can think of switching to another protocol, using some kind of TCP-sockets. Then you can write the scheduler more efficient with select instead of threads, as threads are also slow and resource consuming. select is a system function, that allows to monitor many socket-connection at once, therefore it scales incredibly efficient with the amount of connections and needs nearly no CPU-power for monitoring.