2

I'm currently coding a chatbot for my streaming. Since it needs to do multiple things at once I'm using the multiprocessing module, that way it can still respond to commands and do functions at the same time. My problem now is that I have one process dedicated to some web scraping and another one to look at chat and respond if a command is being typed. My thoughts were, if I append the information from one process to a global list, and then when the command is being typed in chat, the other process can use the information in the appended list. Well, this didn't work and I learned that this is because the 2 processes don't have shared memory, although both having access to the same list, they are both copies of the list, so even if one appends, in the other process' case, it will still be empty. I've come across a few questions regarding this here on stack overflow, but the examples are very specific and since I'm fairly new to coding still, I had a hard time figuring out how to apply it to my own code. For this exact reason, I've simplified the problem so it can help others who are in a similar situation, by having my example broad enough and simple enough for anyone to understand it once they read the solution. Thus this is not the code I'm actually using for my chatbot, but one that mimics the problem.

import multiprocessing as mp
import time

globalList = []

def readList():
    while True:
        time.sleep(2)
        if globalList:
            print(globalList)
        else:
            print("List is Empty")
            print(globalList)
def writeList():
    while True:
        time.sleep(3)
        globalList.append("Item")
        print(globalList)

if __name__ == '__main__':
    p1 = mp.Process(target=readList)
    p2 = mp.Process(target=writeList)
    p1.start()
    p2.start()

When running this code you can see that the writeList function will keep adding another item to the list, but the readList function will keep showing an empty list.

I hope some master wiz out there can help me with this problem.

Benji
  • 61
  • 6

2 Answers2

1

In Python processes cannot straightforwardly access global mutable objects created by other processes. For this, you can use, for example, a multiprocessing.Manager and its proxy objects. Your adapted example:

import multiprocessing as mp
import time


def readList(shared_list):
    while True:
        time.sleep(2)
        if shared_list:
            print(shared_list)
        else:
            print("List is Empty")
            print(shared_list)


def writeList(shared_list):
    while True:
        time.sleep(3)
        shared_list.append("Item")
        print(shared_list)


if __name__ == '__main__':
    manager = mp.Manager()
    shared_list = manager.list()
    p1 = mp.Process(target=readList, args=(shared_list,))
    p2 = mp.Process(target=writeList, args=(shared_list,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
alex_noname
  • 26,459
  • 5
  • 69
  • 86
  • Thank you for your input. It works, glad you came in and gave your input, otherwise I would have thought that this was impossible. Now that it works with the example I'll try to apply it to my real code. – Benji Nov 13 '20 at 08:45
  • I'm sitting here tinkering and it got me thinking, can I only have one list like this? Or will python understand that let's say list2 = manager.list() isn't a new naming of the previous list? – Benji Nov 13 '20 at 08:54
  • list2 will be the new list and not a link to the previous one – alex_noname Nov 13 '20 at 08:56
0

You can not have that by normal means. Processes have their own memory space. Threads, on the other hand, have same memory space and are ran within one process.

For more, please, reffer to this answer Multiprocessing vs Threading Python

go2nirvana
  • 1,598
  • 11
  • 20
  • Hi there Thanks for your answer, to my understanding, threading work much like asyncio, meaning it works while the other function is waiting. That's great if you don't need one function to continuously run, unfortunately I need this. Since the chat part of the script needs to receive as well as answer incoming chat feed, while the other function sometimes work 5 minutes before it waits allowing the next function to take its place. So what I'm getting at, you're saying it can't be done by "normal means" what would be the irregular means to achieve it then? :P – Benji Nov 10 '20 at 09:45
  • @Benji I'll try to summ it up. Threads are used for IO-bound tasks, multiprocessing is used for CPU-bound tasks. If your app is frozen, because there is a long request to weather API - use threads. If you do a lot of CPU-heavy calculations - use multiprocessing. Irregular means would be to use some kind of external storage to hold common data. Which exactly - i don't know, that's just a concept in my head. – go2nirvana Nov 10 '20 at 09:50
  • Thanks @go2nirvana I'll try and research that to see if I can make it work. – Benji Nov 10 '20 at 10:05