function initialize and object initialization (multiprocessing)

Question

I recently saw a answer/comment about how functions are objects in python. So, I wondering why when I take that example, and create a class around it while initializing a variable, it doesn't work the same way. (The class example receives a pickling error): PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Does anyone know why this is?

Example code from the link:

import multiprocessing as mp

def f(x):
    f.q.put('Doing: ' + str(x))
    return x*x

def f_init(q):
    f.q = q

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None, f_init, [q])
    results = p.imap(f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

Same example while puttin f into a class:

import multiprocessing as mp

class F:
    def __init__(self, q):
        self.q = q
    def f(x):
        self.q.put('Doing: ' + str(x))
        return x*x

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None)
    f = F(q)
    results = p.imap(f.f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

It would be helpful if you indicated in what way the second snippet behaves differently. — cdhowie, Apr 10 '13 at 17:33

unutbu · Accepted Answer · 2013-04-10T19:12:25.617

Instance methods are not automatically picklable. So

p.imap(f.f, jobs)

fails because p.imap tries to pickle the arguments. There is a way to "teach" pickle how to pickle instance methods (see Steven Bethard's answer), but your code has another problem: Passing the queue to the instance leads to a RuntimeError:

RuntimeError: Queue objects should only be shared between processes through inheritance

The error message is a little confusing (at least to me) since you can pass the queue as an argument to p.imap, but you can not pass it to the class F first, and then transfer it to the worker processes through f.f.

Anyway, because of these problems, I'd suggest sticking with the original code instead of trying to wrap the code in a class.

Here is an example of how to pickle instance methods:

import multiprocessing as mp
import copy_reg
import types

def _pickle_method(method):
    # Author: Steven Bethard (author of argparse)
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-
    # instancemethods
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_class
    cls_name = ''
    if func_name.startswith('__') and not func_name.endswith('__'):
        cls_name = cls.__name__.lstrip('_')
    if cls_name:
        func_name = '_' + cls_name + func_name
    return _unpickle_method, (func_name, obj, cls)


def _unpickle_method(func_name, obj, cls):
    # Author: Steven Bethard
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-
    # instancemethods
    for cls in cls.mro():
        try:
            func = cls.__dict__[func_name]
        except KeyError:
            pass
        else:
            break
    return func.__get__(obj, cls)

# This call to copy_reg.pickle allows you to pass methods as the first arg to
# mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in
# PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
# __builtin__.instancemethod failed

copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

class F(object):
    def f(self, x):
        fq.put('Doing: ' + str(x))        
        return x*x

def f_init(q):
    # http://stackoverflow.com/a/3843313/190597 (Olson)
    global fq
    fq = q

def main():
    jobs = range(1,6)
    q = mp.Queue()
    p = mp.Pool(None, f_init, [q]) 
    f = F()
    results = p.imap(f.f, jobs)
    p.close()

    for r in results:
        print(r, q.get())

if __name__ == '__main__':
    main()

yields

(1, 'Doing: 2')
(4, 'Doing: 3')
(9, 'Doing: 4')
(16, 'Doing: 1')
(25, 'Doing: 5')

Thanks for the answer! So does this mean that *every* `multiprocessing.Process()` (or `Pool()`, `imap()`, etc.) ***has*** to be a global function, and not in an object? — chase, Apr 10 '13 at 17:43
Not exactly. The `mp.Process()` or `Pool()` could be defined in a local scope, but the **arguments** to `Process` or `imap`, etc. have to be **picklable**. Here is a list of the [things which are automatically picklable](http://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled). — unutbu, Apr 10 '13 at 17:47
Ok thanks @unutbu ! I was just wondering how this all worked since I figured it would be very desired and common to have several processes on functions from different objects which write to a queue to update a GUI. I knew that the objects passed in a queue had to be picklable, but I didn't realize how you could pass stuff through a queue if the object.function sending things didn't have the queue to write to. — chase, Apr 10 '13 at 17:57

function initialize and object initialization (multiprocessing)

1 Answers1

Linked