-1

I wrote the following helper function to run arbitrary functions in parallel.

import multiprocessing


def runParallel(fns=[], args=[]):
    print('Starting multiprocessing with %i cores' % (multiprocessing.cpu_count() - 1))
    pool = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1)
    for fn, arg in zip(fns, args):
        pool.apply_async(fn, (arg,))
    pool.close()
    pool.join()

I call the function with an itertools.repeat call of a function and a list of filenames

runParallel(itertools.repeat(self.processFile), fileNamesAndPaths)

processFile is a classmethod with the signature

def processFile(self, filename):

and starts with a 'print'-statement which is never executed. The programm just ends after the output "starting multiprocessing with 3 cores".

using Process from multiprocessing works in general but it floods my CPU with an amount of processes it can't handle and freezes eventually but at least the processFile function is called

from multiprocessing import Process


def runParallel(fns=[], args=[]):
    proc = []
    for fn, arg in zip(fns, args):
        p = Process(target=fn, args=(arg,))
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

This is why I wanted to use pool since from my understanding it would handle the amount of processes at any given time.

If it is helpful I run this with 2.7.10 on a 64 bit Windows machine.

NoMonkey
  • 3
  • 3
  • Are you saying it works with `Process` and `fn` == an instance method? – Peter Wood May 04 '17 at 07:09
  • @PeterWood Yes it does but as I have roughly 300 files in the list the program creates 300 processes which crashes my pc after 20 seconds. – NoMonkey May 04 '17 at 09:25

1 Answers1

-1

You could replace apply_async with apply to see the error output. Btw, it works in python3, I'm not sure about python2.7.

Sraw
  • 18,892
  • 11
  • 54
  • 87
  • Don't know how to format the code properly in a comment so I just post the last line: `cPickle.PicklingError: Can't pickle : attribute lookup __builtin__.instancemethod failed` Which suggests that Pool can't handle class methods? – NoMonkey May 04 '17 at 07:03
  • @NoMonkey http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma still btw, you can post your code by editing your question. – Sraw May 04 '17 at 07:07
  • Thanks for the link. Within I found this [link](http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma/7309686#7309686) which helped me with the problem – NoMonkey May 04 '17 at 13:36
  • @NoMonkey What was the problem? – Peter Wood May 04 '17 at 19:17
  • @PeterWood Apparently for Pool from multiprocessing the functions that you want to call parallel need to be pickled. As far as I understand python 2.7 can't pickle class methods by default so you have do some extra work described in the link i provided in the last comment. – NoMonkey May 05 '17 at 11:43