Adding to a list in a running python process

Question

I have a class Feed that inherits from multiprocessing.Process, it has a list within it. While the processes are running I need to be able to add to this list. Is this possible?

class Feed(multiprocessing.Process):
    def __init__(self, user: str, feed_id: str, feed_name: str):
        super(Feed, self).__init__()

        self._user = user
        self._feed_id = feed_id
        self._feed_name = feed_name

        self._connected_servers_store = []


    def add_server(self, server_ip: str) -> None:
        self._connected_servers_store.append(server_ip)

    def run(self) -> None:
        while True:
            print(self._connected_servers_store)

I would like to be able to do this but haven't been able to find anything online that deals with this type of scenario. For the purposes of this application, the IP addresses cant be hardcoded in.

def main():
    feed = Feed(user='1234', feed_id='12345', feed_name='Feed 1')

    feed.start()

    feed.add_server('127.0.0.1')
    feed.add_server('<IP_ADDRESS>')
    feed.add_server('<IP_ADDRESS>')

if __name__ == '__main__':
    main()

When the program is running all that is printed is an empty list.

Is there a problem with the code you have tried / posted here? — Hymns For Disco, Apr 14 '20 at 08:41
If you are meaning a problem with the code in the question then yeah, it doesn't add the IPs to the list. I should have specified, I will edit the question. — Scott.M, Apr 14 '20 at 08:46
Does this question help you? https://stackoverflow.com/questions/38322574/python-multiprocessing-sharing-of-global-values — Hymns For Disco, Apr 14 '20 at 13:41

Hannu · Accepted Answer · 2020-04-14T12:42:20.287

Your problem is in sharing variables between processes. What happens here is that your process 1 (main process) creates the class object, which then spawns another process 2 where your run function executes. How you have written your code your add_server modifies objects in process 1 space but then you print in process 2.

This can be fixed by using a manager.

Consider this version of your code (irrelevant parts omitted):

def __init__(self, user: str, feed_id: str, feed_name: str):    
    super(Feed, self).__init__()
    self._user = user
    self._feed_id = feed_id
    self._feed_name = feed_name
    self._connected_servers_store = multiprocessing.Manager().list()

def add_server(self, server_ip: str) -> None:
    self._connected_servers_store.append(server_ip)

def run(self) -> None:
    while True:
        print ("---")
        for i in self._connected_servers_store:
            print(i)
        sleep(0.5)

This works now as intended. If my memory serves me right, there are some additional lines of code you need to add if running on Windows (you will get exceptions if these are not there) but I am not a Windows guru and might be wrong.

NB: for this to work you must do something to prevent your main process from exiting. You will get an exception with your current main as your main process exits after adding the IP addresses and the manager is dropped.

This is the quickfix to that in the example code. It is good programming practice anyway to use your main program to ensure subprocesses are shut down orderly and their exception handling is in place. Without sleep call you will get obscure exceptions from Manager.

feed.add_server('127.0.0.1')
feed.add_server('<IP_ADDRESS>')
feed.add_server('<IP_ADDRESS>')
sleep(30)

Adding to a list in a running python process

1 Answers1