Multi processing tasks not running

Question

I have two threads that receive asynchronous data and want to process this data in the main thread. the first process the main thread call (ProcessA) runs but the second is never executed. For ease of space and explanation I've simplified the threads. The first thread is:

import logging
import time
logging.basicConfig(filename='ProcessA.log',level=logging.DEBUG)
from multiprocessing import Queue
q = Queue()
class ProcessA(q):
    global variableA 
    logging.info('ProcessA started') 
    while (True):
        variableA = 'A' + str(time.time()) 
        q.put(variableA)
        logging.info (variableA)
        time.sleep(5)

The second thread is similar:

import time
from multiprocessing import Queue
import logging
logging.basicConfig(filename='ProcessB.log',level=logging.DEBUG)
q = Queue()
class ProcessB(q):
    logging.info('ProcessB started') 
    global variableB 
    while (True):
        variableB = 'B' + str(time.time()) 
        q.put (variableB)
        logging.info (variableB)
        time.sleep(2)

The main calling thread is as follows:

import time 
from multiprocessing import  Process, Queue
import ProcessA.py
import ProcessB.py

import logging
logging.basicConfig(filename='MThread.log',level=logging.DEBUG)

if __name__ == "__main__":
    queue = Queue()
    a =Process(target=ProcessA, args=(queue,))
    a.start()

    b = Process(target=ProcessB, args=(queue,))
    b.start()

    while (True):
        if not queue.empty():
            variableC = queue.get()
            logging.info ("variableC ="+ variableC) 
            time.sleep(1)

When I run the program only the ProcessA runs (as I see in the log file).

I get the same result whether I run the code native in a Windows 7 terminal or in Spyder with 'Run>Configuration ..> Execute in external terminal' set

What can I do to get the ProcessB and the remainer of the main thread to run ?

UPDATE Moving the classes from external files to internal functions works, however using external functions doesn't work.

That is this code works:

import time 
from multiprocessing import  Process, Queue
import logging
logging.basicConfig(filename='MThread2.log',level=logging.DEBUG)

def ProcessA(q):
    global variableA 
    logging.info('ProcessA started') 
    while (True):
        variableA = 'A' + str(time.time()) 
        q.put(variableA)
        logging.info (variableA)
        time.sleep(5)

def ProcessB(q):
    logging.info('ProcessB started') 
    global variableB 

    while (True):
        variableB = 'B' + str(time.time()) 
        q.put (variableB)
        logging.info (variableB)
        time.sleep(2)


if __name__ == "__main__":
    queueA = Queue()
    a =Process(target=ProcessA, args=(queueA,))
    a.start()
    queueB = Queue()
    b = Process(target=ProcessB, args=(queueB,))
    b.start()

    while (True):
        if not queueA.empty():
            variableC = queueA.get()
            logging.info ("variableC ="+ variableC) 
        if not queueB.empty():
            variableC = queueB.get()
            logging.info ("variableC ="+ variableC) 


        time.sleep(1)

However moving the code into external functions still doesn't work either.

score 0 · Answer 1 · answered Jun 18 '18 at 20:22

It's hard to tell exactly what is wrong here.. I see some small errors with importing and assigning your classes...

To put you on the right path:

ProcessA and B classes could just be functions, which have Queue as an argument, if you would like to do it that way. Otherwise, it should inherit Process (not q) and take queue in on initialization. Here's a good walkthrough (Queues and Pipes are about halfway down): https://www.geeksforgeeks.org/multiprocessing-python-set-2/

You might also try using a manager to share this state as well. See https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes

score 0 · Answer 2 · answered Jun 19 '18 at 07:46

The problem was the way in which I was importing the external functions. I didn't need the '.py' extension. I found this out from this post: Call a function from another file in Python

The following code works:

import time 
from multiprocessing import  Process, Queue
from ProcessAF import processA
from ProcessBF import processB 


import logging
logging.basicConfig(filename='MThread.log',level=logging.DEBUG)

if __name__ == "__main__":

     queueA = Queue()
     a =Process(target=processA, args=(queueA,))
     a.start()

     queueB = Queue()
     b = Process(target=processB, args=(queueB,))
     b.start()

     while (True):
        if not queueA.empty():
            variableC = queueA.get()
            logging.info ("variableC ="+ variableC) 
            if not queueB.empty():
                variableC = queueB.get()
                logging.info ("variableC ="+ variableC) 


            time.sleep(1)

with a function in the file called ProcessAF.py in the same directory as follows:

import logging
import time
logging.basicConfig(filename='ProcessA.log',level=logging.DEBUG)
from multiprocessing import Queue
q = Queue()
def processA(q):

    global variableA 
    logging.info('ProcessA started') 
    while (True):
        variableA = 'A' + str(time.time()) 
        q.put(variableA)
        logging.info (variableA)
       time.sleep(5)

Multi processing tasks not running

2 Answers2