How to return multiple values from multiprocessing pool as a dataframe?

Question

I have an array of orders of parameters pqd for arima like:

[(0, 0, 0),
 (0, 1, 0),
 (0, 2, 0),
 (0, 3, 0),
 (1, 0, 0),
 (1, 1, 0),
 (1, 2, 0),
 (1, 3, 0),
 (2, 0, 0),
 (2, 1, 0),
 (2, 2, 0),
 (2, 3, 0)]

I want to eval arima's performance using that order of parameters using multiprocessing. however I have two issues:

The eval function takes 3 parameters
The eval function returns 2 results

How do I return results for both error and model as a dataframe?

Here is what I have tried:

from multiprocessing import Pool
#this computes error and return a model
def eval_model_parameters(a,b,order):
    #use order PARAMETER somehow to compute model 
    error = a*b/2
    model = a
    return [error,model]

p = [0, 1 , 2]
d = [0, 1 , 2 ,3]
q = [0]
pdq = list(itertools.product(p, d, q)) 

p = Pool(7)
func = eval_model_parameters(1, 2, pdq)
res  = p.map(func, pdq)  
p.close()

I tried doing this to pass parameters to function

func = eval_model_parameters(1, 2, pdq)

And this to return results

res  = p.map(func, pdq)

but I get

---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "Anaconda3\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
TypeError: 'list' object is not callable
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
<ipython-input-53-7743fd9ca60b> in <module>
     13 p = Pool(7)
     14 func = eval_model_parameters(1, 2, pdq)
---> 15 res  = p.map(func, pdq)
     16 p.close()
     17 

Anaconda3\lib\multiprocessing\pool.py in map(self, func, iterable, chunksize)
    266         in a list that is returned.
    267         '''
--> 268         return self._map_async(func, iterable, mapstar, chunksize).get()
    269 
    270     def starmap(self, func, iterable, chunksize=None):

Anaconda3\lib\multiprocessing\pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

TypeError: 'list' object is not callable

What is the correct way to acomplish this?

score 1 · Answer 1 · answered May 20 '20 at 05:33

1

The first argument to the pool.map function should be a callable object. In your case, you have called the function yourself and then passing the results to pool.map function. Try passing the function itself to the pool.map like below:

p.map(eval_model_parameters, pdq)

Now when you change the above line and run the code you will see that the tuple of pdq list is getting passed as a single argument. To solve that please follow this question multiple args to pool.map

Hope this helps!!

answered May 20 '20 at 05:33

HNMN3

542
3
9

Understand it, but what about parameters a and b ? – edgarmtze May 20 '20 at 05:35
they will be the first and second values in the tuple of pdq list. – HNMN3 May 20 '20 at 05:36
could you please elaborate it?, still dont get the idea – edgarmtze May 20 '20 at 05:36
The list pdq consist of tuple each having three values, when we map some function with this tuple then these three values will be passed as arguments to that function. – HNMN3 May 20 '20 at 05:40
Ok, last question is how to return multiple values from the function that evaluates, maybe a list or something like that? – edgarmtze May 20 '20 at 05:43
You are already returning a list of values in eval_model_parameters function, right? – HNMN3 May 20 '20 at 05:46
so in the line res = p.map(func, pdq) res has a list of lists of results? – edgarmtze May 20 '20 at 05:48
yes the variable "res" is list of lists, having two values returned from the function. – HNMN3 May 20 '20 at 05:51

How to return multiple values from multiprocessing pool as a dataframe?

1 Answers1