RecursionError: maximum recursion depth exceeded while calling a Python object when using pickle.load()

Question

Firstly I'm aware that there have been multiple questions already asked regarding this particular error but I can't find any that address the precise context in which it's occurring for me. I've also tried the solutions provided for other similar errors and it hasn't made any difference.

I'm using the python module pickle to save an object to file and the reload it using the following code:

with open('test_file.pkl', 'wb') as a: 
    pickle.dump(object1, a, pickle.HIGHEST_PROTOCOL)

This doesn't throw any error but then when I try and open the file using the following code:

with open('test_file.pkl', 'rb') as a:
    object2 = pickle.load(a)

I get this error:

---------------------------------------------------------------------------

RecursionError                            Traceback (most recent call last)
<ipython-input-3-8c5a70d147f7> in <module>()
      1 with open('2test_bolfi_results.pkl', 'rb') as a:
----> 2     results = pickle.load(a)
      3

~/.local/lib/python3.5/site-packages/elfi/methods/results.py in __getattr__(self, item)
     95     def __getattr__(self, item):
     96         """Allow more convenient access to items under self.meta."""
---> 97         if item in self.meta.keys():
     98             return self.meta[item]
     99         else:

... last 1 frames repeated, from the frame below ...

~/.local/lib/python3.5/site-packages/elfi/methods/results.py in __getattr__(self, item)
     95     def __getattr__(self, item):
     96         """Allow more convenient access to items under self.meta."""
---> 97         if item in self.meta.keys():
     98             return self.meta[item]
     99         else:

RecursionError: maximum recursion depth exceeded while calling a Python object

I'm aware other people have seen this same error (Hitting Maximum Recursion Depth Using Pickle / cPickle) when doing pickle.dump and I've tried increasing the maximum recursion depth by doing sys.setrecursionlimit() but this doesn't work, I either get the same error as above or I increase it further and python crashes with the message: Segmentation fault (core dumped).

I suspect that the root of the problem is actually when I save the object with pickle.load() but I don't really know how to diagnose it.

Any suggestions?

(I'm running python3 on a windows 10 machine)

Answer 1

Here's a fairly minimal class derived from collections.UserDict which performs the same trick that your problem object does. It's a dictionary which allows you to access its items either via normal dict syntax, or as attributes. I've thrown in a few print calls so we can see when the main methods get called.

import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

# test

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

output

SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3

---

So far, so good. But if we try to pickle our d instance, we get RecursionError because of that __getattr__ which does the magic conversion of attribute access to key lookup. We can overcome that by providing the class with __getstate__ and __setstate__ methods.

import pickle
import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

    def __getstate__(self):
        print('GS')
        return self.data

    def __setstate__(self, state):
        print('SS')
        self.data = state

# tests

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)

print('Unpickling')
obj = pickle.loads(s)
print(obj)

output

SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3
Pickling
GS
b'\x80\x04\x95D\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x08AttrDict\x94\x93\x94)\x81\x94}\x94(\x8c\x04zero\x94K\x00\x8c\x03one\x94K\x01\x8c\x03two\x94K\x02\x8c\x05three\x94K\x03ub.'
Unpickling
SS
SA data {'zero': 0, 'one': 1, 'two': 2, 'three': 3}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}

But what can we do to repair an existing class with this behaviour? Fortunately, Python allows us to easily add new methods to an existing class, even one that we obtain via importing.

import pickle
import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

# Patch the existing AttrDict class with __getstate__ & __setstate__ methods

def getstate(self):
    print('GS')
    return self.data

def setstate(self, state):
    print('SS')
    self.data = state

AttrDict.__getstate__ = getstate
AttrDict.__setstate__ = setstate

# tests

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)

print('Unpickling')
obj = pickle.loads(s)
print(obj)

This code produces the same output as the previous version, so I won't repeat it here.

Hopefully, this gives you enough info to repair your faulty object. My __getstate__ & __setstate__ methods only save and restore the stuff in the .data dictionary. To properly pickle your object, we may need to be a bit more drastic. For example, we may need to save and restore the instance's .__dict__ attribute, rather than just the .data attribute, which corresponds to the .meta attribute in your problem object.