0

This question is related to these other posts on SO, yet the solutions suggested therein do not seem to work for my case. In short, my problem can be illustrated by the following example. I have an Algebra class where by the method triPower I aim at computing the power of a trinomial, i.e. (a+b+c)**n for many n values with fixed a, b, c. To do so, I thought of creating a method _triPower(a,b,c,n) and pass it to my pool.map() function by functools.partial(_triPower,...) where I fix a, b, c and leave n as the only parameter, since I am working in Python 2.7 and map from the multiprocessing module wants only one argument function (see otherwise this post). The code is the following:

from __future__ import division
import numpy as np
import functools as fntls
import multiprocessing as mp
import multiprocessing.pool as mppl

# A couple of classes introduced to allow multiple processes to have their own daemons (parallelization)
class NoDaemonProcess(mp.Process):
   # make 'daemon' attribute always return False
   def _get_daemon(self):
      return False
   def _set_daemon(self, value):
      pass
   daemon = property(_get_daemon, _set_daemon)

# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(mppl.Pool):
    Process = NoDaemonProcess

# Sample class where I want a method to run a parallel loop
class Algebra(object):
    def __init__(self,offset):
        self.offset = offset

    def trinomial(self,a,b,c):
        return a+b+c

    def _triPower(self,a,b,c,n):
    """This is the method that I want to run in parallel from the next method"""
        return self.offset + self.trinomial(a,b,c)**n

    def triPower(self,n):
        pls = MyPool(4)
        vals = pls.map(fntls.partial(self._triPower,a=1.,b=0.,c=1.),n)
        print vals

# Testing
if __name__ == "__main__":
    A = Algebra(0.)
    A.triPower(np.arange(0.,10.,0.01))

The above does not work and produces (as expected from this post) the error: 

cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Hence, following the same post, I tried to define _triPower as a global function i.e.

def _triPower(alg,a,b,c,n):
"""This is the method that I want to run in parallel from the next method"""
    return alg.trinomial(a,b,c)**n

and then editing Algebra.triPower(...) according to:

    def triPower(self,n):
    pls = MyPool(4)
    vals = pls.map(fntls.partial(_triPower, alg=self, a=1., b=0., c=1.), n)
    print vals

and this latter instead gives some weird TypeError like:

TypeError: _triPower() got multiple values for keyword argument 'alg'

On the other hand the suggestion to try to make the methods serializable by VeryPicklableObject as in this other post seems also not to work and this package appears dead by now (as of 05/2019). So what am I doing wrong and how can I make my computation run in parallel?

maurizio
  • 745
  • 1
  • 7
  • 25
  • 1
    In the "partial" call use simple position args like "self, 1., 0., 1." instead of keyword args. – Michael Butscher Jun 01 '19 at 18:55
  • @MichaelButscher Thanks, it solved the issue. I would however like to understand why it is like that and the `self` variable is not recognized. Also what if the variable that I want to vectorize/parallelize for is not `n` but any others with position not at the end of the list of input arguments of `_triPower(alg,a,b,c,n)`, i.e. say for example `b`? I don't want to rewrite `_triPower` for any possible ordered configuration of inputs. – maurizio Jun 02 '19 at 01:10
  • 1
    The [docs](https://docs.python.org/2/library/functools.html#functools.partial) show functional equivalent Python code for `partial`. The function returned by `partial` first applies given positional arguments at creation (none in your code), then pos. args. at call (here `n` only, which is applied to `alg` parameter). Then keyword arguments from call or creation. As there is also a keyword argument for `alg`, the error occurs. A solution can be instead of `partial` to pack all parameters into a tuple and use that as single argument to `_triPower`. – Michael Butscher Jun 02 '19 at 03:35

0 Answers0