1

How can I pass complete parameters, without splitting them among the cores, to function myfun and at the same time pass the respective element from the collection [1,2,3,4,5]

p=Pool(5)
p.map(myfun(df_A,df_B),[1,2,3,4,5])

If I implement it this way, the function gets the parameters df_A and df_B but not an element from the collection

Here is an example how myfun can look like:

def myfunc(df_A, df_B, e):
    do_something
    print(df_A.iloc[e],df_A.iloc[e])

e is one element of the collection [1,2,3,4,5]

Julio Reckin
  • 146
  • 1
  • 10
  • Can you show the code for *myfun* – DarkKnight Dec 15 '21 at 17:03
  • It's just an example, the aim is to have the parameters df_A, df_B and one element of the collection in myfun, no matter how the function looks like – Julio Reckin Dec 15 '21 at 17:08
  • 1
    You are not supposed to call the function in the `map` call. You are supposed to just ***pass*** it. And depends on how your function is defined you might need to use [`starmap`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap) – Tomerikoo Dec 15 '21 at 17:08
  • 1
    It does matter how the function looks like, or more accurately, how its ***signature*** looks like... – Tomerikoo Dec 15 '21 at 17:09
  • Does this answer your question? [How to use multiprocessing pool.map with multiple arguments](https://stackoverflow.com/questions/5442910/how-to-use-multiprocessing-pool-map-with-multiple-arguments) – Tomerikoo Dec 15 '21 at 17:10
  • Use partial application – juanpa.arrivillaga Dec 15 '21 at 17:25

1 Answers1

2

You could consider something like this (there are probably better ways):

from multiprocessing import Pool

def myfunc(a, b, c):
    print(a, b, c)

df_A = 1
df_B = 2

def main():
    with Pool() as pool:
        pool.starmap(myfunc,[[df_A, df_B, x] for x in range(1, 6)])

if __name__ == '__main__':
    main()

Output:

1 2 1
1 2 2
1 2 3
1 2 4
1 2 5
Tomerikoo
  • 18,379
  • 16
  • 47
  • 61
DarkKnight
  • 19,739
  • 3
  • 6
  • 22