serialize / pickle a function that is defined in a string

Question

I want to do the following:

import pickle
namespace = {}
exec('def f(x): return x', namespace)
pickle.dumps(namespace['f'])

However, this results in the following error:

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-102-61493bb3c732> in <module>()
      2 namespace = {}
      3 exec('def f(x): return x', namespace)
----> 4 pickle.dumps(namespace['f'])

PicklingError: Can't pickle <function f at 0x7f2134171950>: it's not the same object as __main__.f

The problem that I want to solve: I have a function in string format and I need to be able to serialize it (for parallelization purposes.)

You should explain what is currently happening in your code that is not meeting expectation. What happens when that code runs right now. What do you expect should happen. What *exactly* are you trying to achieve with your code? Please revise how to put together a more solid [mcve] to help get your question across better. — idjaw, Jan 21 '17 at 17:56
Thank you for your comment, I have added the error message. Furthermore, I think that I describe my problem quite clearly. — johnbaltis, Jan 21 '17 at 17:59

score 2 · Answer 1 · edited May 23 '17 at 12:00

2

You can just pass the current global namespace to exec, then you can pickle the function:

import pickle
exec('def f(x): return x', globals())
pickle.dumps(f)

Whether you can then load it in another environment, is an entirely different question as discussed here.

edited May 23 '17 at 12:00

Community

1
1

answered Jan 21 '17 at 18:05

user2390182

72,016
6
67
89

score 2 · Accepted Answer · answered Jan 23 '17 at 07:12

I'm not sure why use used the dill tag in your question, because you only used pickle... but if you really need your code to be exactly how you wrote it, with the one caveat that you can replace pickle with dill... it works:

>>> import dill as pickle
>>> namespace = {}
>>> exec('def f(x): return x', namespace)
>>> _f = pickle.dumps(namespace['f'])
>>> _f
'\x80\x02cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01U\x08CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x01K\x01KCU\x04|\x00\x00Sq\x05N\x85q\x06)U\x01xq\x07\x85q\x08U\x08<string>q\tU\x01fq\nK\x01U\x00q\x0b))tq\x0cRq\r}q\x0e(U\x0c__builtins__q\x0fc__builtin__\n__dict__\nh\nh\x00(h\rh\x0eh\nNN}q\x10tq\x11Rq\x12uh\nNNh\x10tq\x13R0h\x12.'
>>> f = pickle.loads(_f)
>>> f(5)
5
>>>

score 1 · Answer 3 · answered Jan 21 '17 at 18:16

Without using globals() the following works as well:

import pickle
namespace = {}
exec('def f(x): return x', namespace)
f = namespace['f'] # has to have the same name
pickle.dumps(f)

The function taken from namespace has to be defined in the function namespace under the same name.

serialize / pickle a function that is defined in a string

3 Answers3