0

I used to use this snippet to monitor the progress of of a multiprocess. Though, now q.qsize() stopped working and only returns zeros. Is there a replacement for what q.qsize() used to do formerly? Or is there a better way of doing that?

from multiprocessing import Pool, Manager, cpu_count

def process(args):
    '''
    Pickleable function for (parallel) processing.
    Expects a dictionary with keys:
        Mandatory:
        - 'filename': 'path to file to be processed',
        ...
        Optional:
        - 'queue': instance of multiprocessing.Manager().Queue()
    '''
    filename = args['filename']
    mode = args['mode']

    if 'queue' in args.keys():
        q = args['queue']
        q.put('filename')
return do_something_with(filename)
...


pool = Pool(processes=nthreads)
m = Manager()
q = m.Queue()

# prepare args...

results = pool.map_async(process, args)

# monitor progress
while True:
    if results.ready():
        break
    else:
        size = q.qsize()
        self.progress = int(100 * (size / self.n_files))
        time.sleep(1)
self.progress = 100
Soerendip
  • 7,684
  • 15
  • 61
  • 128
  • Actually, I think this works. I had a bug in my code. And there was some code missing here that I added. But it is probably not the best way of doing this. The aim was to use process for serial and parallel cases. – Soerendip Feb 10 '20 at 23:39

1 Answers1

0

It looks like you're mixing up your Pools and Queues. The multiprocessing Pool class handles the queuing logic for you but the Pool is not a Queue object, it's a Pool. So you're when you're checking your Queue size, there simply isn't anything there.

To directly answer your question though, your q.qsize() is always going to be 0 because you're not actually enqueuing anything to your Queue. Your queue is always empty. You need to q.put() something in your queue, if that is the behavior you are looking for.

Nothing that I know of will work exactly with your code as you have it right now with pool.map_async. I'm not sure how you want to approach it from here, but there are lots and lots of questions about checking how many items are left in a Pool.

C. Fennell
  • 992
  • 12
  • 20