Multiprocessing Python with RPYC "ValueError: pickling is disabled"

Question

I am trying to use the multiprocessing package within an rpyc service, but get ValueError: pickling is disabled when I try to call the exposed function from the client. I understand that the multiprocesing package uses pickling to pass information between processes and that pickling is not allowed in rpyc because it is an insecure protocol. So I am unsure what the best way (or if there is anyway) to use multiprocessing with rpyc. How can I make use of multiprocessing within a rpyc service? Here is the server side code:

import rpyc
from multiprocessing import Pool

class MyService(rpyc.Service):

    def exposed_RemotePool(self, function, arglist):

        pool = Pool(processes = 8)
        result = pool.map(function, arglist)
        pool.close()
        return result


if __name__ == "__main__":
    from rpyc.utils.server import ThreadedServer
    t = ThreadedServer(MyService, port = 18861)
    t.start()

And here is the client side code that produces the error:

import rpyc

def square(x):
    return x*x

c = rpyc.connect("localhost", 18861)
result = c.root.exposed_RemotePool(square, [1,2,3,4])
print(result)

http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma?rq=1 — Peter Teoh, Nov 13 '14 at 01:34
The object can ordinarily be pickled (it's just an integer), but rpyc forbids pickling and thus (seemingly) multiprocessing which uses pickling. I am not implementing a new protocol, I am trying to use rpyc. If it cannot be done I will look for other solutions but I would rather not reinvent the wheel. And if it cannot be done I am at a loss for why rpyc exists, i.e. what's the point of using another machine if you can only make use of one processor on that multi-core machine. — Michael, Nov 13 '14 at 01:42
there are many reasons for multithreading not easily done in a network env. one of them is that CPU assignment is done inside the kernel, and userspace thread library can only request to use this facility - but not always guaranteed. And well said is this: http://intermediate-and-advanced-software-carpentry.readthedocs.org/en/latest/multiprocessing.html (that GIL makes multithreaded as fast as single threaded, and that "global" lock is within a single host). — Peter Teoh, Nov 13 '14 at 03:55
@shx2 it's unclear how to pass that setting to the service (see comment on answer below). — Michael, Nov 15 '14 at 01:19
Btw, this also happens when using copy.deepcopy() which also uses pickle in its implementation, not just with multiprocessing — Pelle, Jan 19 '16 at 13:50

score 5 · Accepted Answer · answered Nov 26 '14 at 00:29

You can enable pickling in the protocol configuration. The configuration is stored as a dictionary, you can modify the default and pass it to both the server (protocol_config = ) and client (config =). You also need to define the function being parallelized on both the client and server side. So here is the full code for server.py:

import rpyc
from multiprocessing import Pool
rpyc.core.protocol.DEFAULT_CONFIG['allow_pickle'] = True

def square(x):
    return x*x


class MyService(rpyc.Service):

    def exposed_RemotePool(self, function, arglist):

        pool = Pool(processes = 8)
        result = pool.map(function, arglist)
        pool.close()
        return result



if __name__ == "__main__":
    from rpyc.utils.server import ThreadedServer
    t = ThreadedServer(MyService, port = 18861, protocol_config = rpyc.core.protocol.DEFAULT_CONFIG)
    t.start()

And for client.py the code is:

import rpyc

rpyc.core.protocol.DEFAULT_CONFIG['allow_pickle'] = True

def square(x):
    return x*x

c = rpyc.connect("localhost", port = 18861, config = rpyc.core.protocol.DEFAULT_CONFIG)
result = c.root.exposed_RemotePool(square, [1,2,3,4])
print(result)

score 1 · Answer 2 · answered Nov 13 '14 at 09:28

1

You should enable it in the protocol configuration. See http://rpyc.readthedocs.org/en/latest/api/core_protocol.html#rpyc.core.protocol.DEFAULT_CONFIG

answered Nov 13 '14 at 09:28

sebulba

1,245
2
10
17

I added `rpyc.core.protocol.DEFAULT_CONFIG['allow_pickle'] = True` to my code at the top, but I am still getting the error. Unsure how to change this setting for the ThreadedServer object. – Michael Nov 14 '14 at 21:08

Multiprocessing Python with RPYC "ValueError: pickling is disabled"

2 Answers2

Linked