Cannot pickle object: maximum recursion depth exceeded

Question

I'm trying to pickle objects that I generate in a script to process them afterwards, but I get this error:

File "<ipython-input-2-0f716e86ecd3>", line 1, in <module>
pickle.dump(thisperson, output, pickle.HIGHEST_PROTOCOL)
RecursionError: maximum recursion depth exceeded while pickling an object

Here thisperson is relatively simple (or so I thought): a name and a dictionary containing attributes:

class Attributes:
    def __init__(self, what_attr, attr):
        self._what_attr = what_attr
        self._attributes = attr

    @property
    def what_attr(self):
        return self._what_attr
    @what_attr.setter
    def what_attr(self, new_value):
        self._what_attr = new_value

    @property
    def attributes(self):
        return self._attributes
    @attributes.setter
    def attributes(self, new_value):
        self._attributes = new_value


class Person:
    def __init__(self, watname, watage):
        self._myname = watname
        self._age = watage
        self._attributes = []

    @property
    def attributes(self):
        return self._attributes
    @attributes.setter
    def attributes(self, attributes_list):
        self._attributes = attributes_list

    @property
    def myname(self):
        return self._myname
    @myname.setter
    def myname(self, value):
        self._myname = value

    @property
    def age(self):
        return self._age
    @age.setter
    def age(self, value):
        self._age = value

the pickle function looks like this:

import pickle

def save_person(person, mypath):
    import os
    if not os.path.exists(mypath):
        os.makedirs(mypath)
    my_path = mypath + str(person.myname)
    if not os.path.exists(my_path + '/'):
        os.makedirs(my_path + '/')
    with open(my_path + '.pkl', 'wb') as output:
        pickle.dump(person, output, pickle.HIGHEST_PROTOCOL)

This error seems to be coming from the fact that the class contains a string and a dict because if I keep attributes as an empty dict I don't get the error.

Is there another way I can pickle objects like these?

I tried increasing sys.setrecursionlimit, but that didn't help.

Edit: .... i can't reproduce the error if i use a short version of the code .... go figure. @MinhNguyen something weird is that i create my attributes dict in a function (attr = get_attributes(data)), but when i create the same dict myself, i can pickle it ....

thisperson = Person("Kenny", 22, {})
attr = get_attributes(data)  # returns {'eyes': 'blue', 'hair': 'brown'}
this_attr = Attributes(person_name, attr)
this_person.attributes.append(this_attr)
save_person(this_person, main_folder)

Exception ignored in: 
'_pydevd_frame_eval.pydevd_frame_evaluator_win32_37_64.get_bytecode_while_frame_eval'
 RecursionError: maximum recursion depth exceeded while calling a Python object
 Process finished with exit code -1073741819 (0xC0000005)

but if i do the following it works:

 attr = {'eyes': 'blue', 'hair': 'brown'}
 this_attr = Attributes(person_name, attr)
 this_person.attributes.append(this_attr)
 save_person(this_person, main_folder)

Solution

Alright so after further digging, i found here that pickle doesnt mingle with dict well and will often crash.

So the workaround was that i dropped the Attribute class completely and now my Person's _attribute is a string of dict self._attributes = str(dict()) and whenever i want to add things to it, i'll do an eval(), update() and str() again.

This seem to be working so far...

As a side note, you probably want to being using single-underscores for attributes, as in `_name` and `_attributes`, rather that double-underscores. That former is a convention for indicating the an attribute is an implementation detail, while the latter [invokes name mangling](https://stackoverflow.com/questions/1301346/what-is-the-meaning-of-single-and-double-underscore-before-an-object-name). — Brian61354270, Sep 13 '20 at 22:20
Since pickle complains about recursion depth exceeded, I don't think your dictionary is causing a problem. Can you provide more source codes? Maybe you implement some recursive functions with depth depending on the content of the dictionary? — Minh Nguyen, Sep 14 '20 at 04:13
@MinhNguyen i edited my post to include more info. I tried checking the attributes dict in terms of memory and types, they are the same whether i use my function or create it manually — jmo, Sep 14 '20 at 17:41
Why are you using `@property` here? NONE of your attributes benefit from it even slightly - it's for situations where the public attributes don't correspond exactly to the instance variables (there's a side-effect on setting something, there's calculation involved in reading a value, etc.). — jasonharper, Sep 14 '20 at 17:56
@jasonharper that's True, but even if i take all properties and setters out of the classes definitions, i get the same error. — jmo, Sep 14 '20 at 19:04

score 3 · Answer 1 · edited Feb 09 '22 at 09:05

Problem

I've been using the concurrent.futures module, particularly the ProcessPoolExecutor class, which uses (kind of under the hood) pickle module to return values to the main process. The issue with this, is that pickle module seems to have a limit on what you can "pickle-ize" when we are talking about nested stuff. In my case, I had a structure like the following:

data = [
    {
        "string_list": [
            ["a1", "a2", ..., "an"],
            ["b1", "b2", ..., "bn"],
            ...
            ["n1", "n2", ..., "nn"],
        ]
    },
    {
        "string_list": [
            ["a1", "a2", ..., "an"],
            ["b1", "b2", ..., "bn"],
            ...
            ["n1", "n2", ..., "nn"],
        ]
    },

Basically, it was a list of dictionaries, which everyone of them contained, a list of string lists. I mean, it was a relatively nested structure. And every time i tried to get the data from the ProcessPoolExecutor's result, it raised a RecursionError: maximum recursion depth exceeded while pickling an object, due to pickle module, which can't (apparently) deal with this level of nesting.

Solution

My solution was similar to the provided in the edited question. I "stringified" the structure before pickling it, and then undone it once I could access the data. Personally, I used json module, which also comes with Python Standard Library.

This was my "experience" dealing with the pickle module.

score 1 · Answer 2 · answered Dec 27 '21 at 13:41

1

Quick remark: I found that for my object, IntEnum caused this. Had one instance of it, replaced it with a bunch of constants, and the exception disappeared.

answered Dec 27 '21 at 13:41

Bert Bril

371
2
12

Cannot pickle object: maximum recursion depth exceeded

Solution

2 Answers2

Problem

Solution