1

In view of randomized structure of my algorithm, I want to run it hundreds of times in parallel. Parameters stay the same every time.

Both of implementations raise some kind of exception (PicklingError, or TypeError).

from multiprocessing import Pool
from itertools import izip, repeat

if __name__ == '__main__':
    G = Class1Object()
    S = Class2Object()

    pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())

    # take 1 tuple input
    # parfun = lambda (G,S): fun(G,S)
    # T = pool.map(parfun, izip(repeat(G, 2), repeat(S, 2)))

    # take 2 arguments
    T = pool.map(fun, [G]*2, [S]*2)

What is the fastest and most convenient way to run function with multiple arguments in parallel?

Bach
  • 6,145
  • 7
  • 36
  • 61
Sergey Ivanov
  • 3,719
  • 7
  • 34
  • 59
  • Doesn't this question address the issue? http://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments – Lucas Fortini Oct 30 '14 at 22:42

1 Answers1

1

The map() function only takes a single iterable. So perhaps you could create an iterable function that returns the different tuples you need (or just create an array of the tuples and pass that).

Be advised that you might have problems in trying to send objects across process boundaries. See my answer here in which I tackled the same issue.

Community
  • 1
  • 1
Multimedia Mike
  • 12,660
  • 5
  • 46
  • 62