1

I am new to Python Multiprocessing. I am deploying my Python code in WSGI Gunicorn Multiprocessing environment. My requirement is, updated value of shared variable by one process needs to be picked up by another process. But I can see though one process is updating the value from 0 to 1 but still other process is not picking up the updated value 1, it is taking old value 0.

I have gone through some of the solution on internet but not able to solve this. Request you to help.

Note: As per my code logic, once lifecycle() method execution gets completed then only get_lifecycle() method will be called. (full code I can't paste due to some restrictions)

import os
from multiprocessing import Value
... some other imports

data = Value('i', 0)

class ResourceDriverHandler:

    def lifecycle(self , driver_files):  (called by a process - say, process 7)
        try:
            logger.info(f'checkpoint1 os.getpid() :{os.getpid()}')
            logger.info('checkpoint1 data.value : %s : ', data.value)
            .... some other code
            driver.operation(driver_files)
            .... some other code
        except DriverConfigError as e:
            logger.info(f'checkpoint2 os.getpid() :{os.getpid()}')
            data.value += 1
            logger.info('checkpoint2 data.value : %s : ', data.value)
            
    def get_lifecycle(self):    (called by another different process - say, process 5, once lifecycle() method execution is over)
        logger.info(f'checkpoint3 os.getpid() :{os.getpid()}')
        logger.info('checkpoint3 data.value : %s : ', data.value)
        .... some other code

Gunicorn WSGI server Logs:

checkpoint1 os.getpid() : 7
checkpoint1 data.value : 0
checkpoint2 os.getpid() : 7
checkpoint2 data.value : 1
checkpoint3 os.getpid() : 5
checkpoint3 data.value : 0  (I was expecting 1 as already data.value is changed from 0 to 1 by another process)
Surya
  • 604
  • 1
  • 6
  • 25
  • This issue is in using a global variable as your shared variable. See this previous question: https://stackoverflow.com/questions/38322574/python-multiprocessing-sharing-of-global-values – blarg Jun 11 '22 at 10:49
  • No, I am using multiprocessing.Value only.. in the examples which you shared they are also using multiprocessing.Value in the same place. – Surya Jun 13 '22 at 07:08
  • In the linked SO question, and in all the examples in the Python docs, the shared variable is passed to the subprocess as an argument. You are trying to make it a global variable and not passing it as an argument. On Windows, each Process has its own copy of the Python environment (virtual machine) and therefore its own set of global variables. So your implementation will not work, because each Process has a separate copy of `data`. That is consistent with what you observe. – Paul Cornelius Jun 13 '22 at 11:12
  • @PaulCornelius thanks for pointing it out. Also I am deploying my code in a container in Kubernetes cluster(Openshift), not in Windows. Anyways, I can't change the code and pass 'data' variable as argument in the sub processes as that particular code change is restricted. Then in my case what should I do? Thanks in advance! – Surya Jun 13 '22 at 13:04
  • There are ways of sending messages from one process to another (Pipe and Queue), and even general methods of passing data around like TCP/IP. You can send a message from `lifecycle` (for instance) and have the receiving end set the global variable. Your requirement, as stated, is literally self-contradictory: it asks to set a shared global variable with a process that (a) does not have such a variable and (b) will not allow you to add one. – Paul Cornelius Jun 13 '22 at 21:01
  • Hi @PaulCornelius, Thanks for your help and guidance. My only requirement is if one variable is updated by one process in lifecycle() method , then if other process calls get_lifecycle() it should get the updated value. I can change code in ResourceDriverHandler class which is having lifecycle() & get_lifecycle() methods but can't change code where these methods are getting called by creating ResourceDriverHandler 's instance. I hope I am now clear :) So can I use Multiprocessing Pipe or Queue in ResourceDriverHandler class and solve it? – Surya Jun 14 '22 at 08:07
  • I can't answer that because I don't know how you are starting your Process. The typical use case for Queue/Pipe is that it gets created in the main Process and passed to the startup function of any subprocess. You need to get a reference to it into your ResourceDriverHandler class somehow. I don't know enough about your program to comment further. TCP/IP will almost always work but is more difficult to implement since it needs a client and a server. – Paul Cornelius Jun 14 '22 at 09:31

0 Answers0