RecursionError when calling pickeled function

Question

I am trying to run the following code:

import pickle

def foo():
    print("i am foo")
    
pickle_foo = pickle.dumps(foo)

def foo():
    print("i am the new foo")
    fkt = pickle.loads(pickle_foo)
    return fkt()

foo()

The expected behavior would be:

the new defined function "foo" is called
in the new function the old function gets unpickeled and then called
output:

i am the new foo
i am foo

What actually happens is:

the new function foo gets called, and then recursively calls itself until a Recursion Error gets thrown:

RecursionError: maximum recursion depth exceeded while calling a Python object

The error does not occur, when the two functions are named differently, but that would be very unpractical for my project. Could anyone explain, why this behavior occurs and how to avoid it (without changing the function names)?

I dont use pickle (so I may be wrong) but it looks like you are calling `pickle_foo` after you call the second `foo` function. That means in order to actually delete the first `foo` you have to put it outside the second `foo` — , Nov 12 '20 at 20:57

score 3 · Answer 1 · answered Nov 12 '20 at 21:08

The pickle module pickles functions based on their fully-qualified name reference. This means that if your function is redefined somewhere in code, and then you unpickle a pickled reference to it, calling it will result in a call to the new definition.

From the Python docs on pickle:

Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. 2 This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.

What you can do however, is use inspect.getsource() to retrieve the source code for your function, and pickle that. This requires that your code be available as source somewhere on the file system, so compiled C code imported, or other outside sources (interpreter input, dynamically loaded modules) will not work.

When you unpickle it, you can use exec to convert it into a function and execute it.

Note: this will redefine foo every time, so calls to foo cannot be guaranteed to have the same effect.

Note 2: exec is unsafe and usually unsuitable for code that will be interacting with external sources. Make sure you protect calls to exec from potential external attacks that attempt to execute arbitrary code.

RecursionError when calling pickeled function

1 Answers1