"Published" value accessible across processes in python

Question

I'm writing software in python (3.7) that involves one main GUI thread, and multiple child processes that are each operating as state machines.

I'd like the child processes to publish their current state machine state name so the main GUI thread can check on what state the state machines are in.

I want to find a way to do this such that if the main process and the child process were trying to read/write to the state variable at the same time, the main process would immediately (with no locking/waiting) get a slightly out-of-date state, and the child process would immediately (with no locking/waiting) write the current state to the state variable.

Basically, I want to make sure the child process doesn't get any latency/jitter due to simultaneous access of the state variable, and I don't care if the GUI gets a slightly outdated value.

I looked into:

using a queue.Queue with a maxsize of 1, but the behavior of queue.Queue is to block if the queue runs out of space - it would work for my purposes if it behaved like a collections.deque and silently made the oldest value walk the plank if a new one came in with no available space.
using a multiprocessing.Value, but from the documentation, it sounds like you need to acquire a lock to access or write the value, and that's what I want to avoid - no locking/blocking for simultaneous read/writes. It says something vague about how if you don't use the lock, it won't be 'process-safe', but I don't really know what that means - what bad things would happen exactly without using a lock?

What's the best way to accomplish this? Thanks!

Why do you have the problem that the Queue runs out of space? Usually you constantly poll the results from the main thread. Because the GUI is usually not so time consuming, it most of the times waits for stuff in the Queue. Did you look into shared variables, e. g. a dictionary? — RaJa, Oct 07 '19 at 10:27
@RaJa I don't want the backend to depend on the front end to poll the Queue in order to work. If the GUI freezes up or something, and the Queue runs out of space, the child process will block until the GUI frees up space! That's not acceptable. I also just don't want to store old values, so for my use case, the Queue is a data container that accumulates unwanted data. — Brionius, Oct 08 '19 at 12:44
That makes sense. Check this: https://stackoverflow.com/a/6832693/4141279 That way you childs can change the values and you GUI just reads them somewhen. No polling. — RaJa, Oct 11 '19 at 12:06
@RaJa Thanks for your help - looks like that would probably work too - I found another solution. — Brionius, Oct 18 '19 at 16:32

score 0 · Accepted Answer · answered Oct 18 '19 at 17:02

For some reason, I had forgotten that it's possible to put into a queue in a non-blocking way!

The solution I found is to use a multiprocessing.Queue with maxsize=1, and use non-blocking writes on the producer (child process) side. Here's a short version of what I did:

Initializing in parent process:

import multiprocessing as mp
import queue

publishedValue = mp.Queue(maxsize=1)

In repeatedly scheduled GUI function ("consumer"):

try:
    # Attempt to get an updated published value
    publishedValue.get(block=False)
except queue.Empty:
    # No new published value available
    pass

In child "producer" process:

try:
    # Clear current value in case GUI hasn't already consumed it
    publishedValue.get(block=False)
except queue.Empty:
    # Published value has already been consumed, no problem
    pass

try:
    # Publish new value
    publishedValue.put(block=False)
except queue.Full:
    # Can't publish value right now, resource is locked
    pass

Note that this does require that the child process can repeatedly attempt to re-publish the value if it gets blocked, otherwise the consumer might completely miss a published value (as opposed to simply getting it a bit late).

I think this may be possible in a bit more concise way (and probably with less overhead) with non-blocking writes to a multiprocessing.Value object instead of a queue, but the docs don't make it obvious (to me) how to do that.

Hope this helps someone.

OK, but didn't you want that the child can update the value any time? In this solution the child process can push a value and has to wait until it is consumed before it can push another one. But if it works for you then its fine. — RaJa, Oct 19 '19 at 16:39
Yeah, you're right, that isn't ideal. However, the key requirement is that the child doesn't **block** - the child process' timing is critical. This solution doesn't block the child process, although the `put` call may fail, requiring the child to republish when it next gets the chance. — Brionius, Oct 21 '19 at 13:44

"Published" value accessible across processes in python

1 Answers1