Python pickle error when dumping list of dictionaries

Question

This is my function:

def pickleIt(file_name,data):
   try:
        with open(file_name+".pickle", "wb") as output_file:
            pickle.dump(data, output_file,-1)
        output_file.close()
    except Exception,e:
        print "Cannot open the file:",file_name,e
    return

It works fine when I pickle a dictionary or a list, however it doesn't work with list of dictionaries. I get a dictionary as a single_record from another function and then append it to my list:

def set_dict(url.....)
     single_record={'url':url,
                    'title':song_obj.getSongTitle(),
                    'name':song_obj.getPerformer(),
                    'author':song_obj.getAuthors(),
                    }
     return single_record

When I try to dump the list of 50 dicts I get the following error:

maximum recursion depth exceeded

Can anyone please help to find whats wrong with that?

Note that doing `except Exception` is a really bad idea that can hide other problems in your code. Instead, catch only *specific* exceptions - in your case, the one thrown if the file can't be opened. — Gareth Latty, Feb 23 '13 at 21:56
Also, when you get an exception, please post the whole stack trace, it makes it significantly easier to work through the problem. — Gareth Latty, Feb 23 '13 at 21:58
And as a final note, when you use the `with` statement to open a file, the context manager handles closing the file for you, you do not need to close it manually. There is another small redundancy in the final `return` - Python functions return at the end regardless, so there is no need to add it there. — Gareth Latty, Feb 23 '13 at 21:59
Thanks,I agree this is not the best way to catch but I still print the error (e) and it says "Maximum recursion depth exceeded",And I cant find the recursion issue here. — Dan can, Feb 23 '13 at 22:03

entropy · Accepted Answer · 2013-02-24T01:49:13.583

The error you are getting "maximum recursion depth exceeded" means that your the call stack is too deep. The depth of the call stack is basically how many functions were called from the initial call point(Something like your main function/loop) without returning.

So for example if from your initial call point you call func1(). In func1() the depth is currently 1. If func1(), before returning calls another function, the depth in that function will be 2.

To find out what the current limit is on the size of the call stack call sys.getrecursionlimit() and to change it, call sys.setrecursionlimit().

If the error is thrown when pickling the code, then chances are, the structure you are trying to pickle is has too many nested elements. This is because, the dump/dumps will recurse on containers in the structure being dumped. Meaning, if you're dumping a list, it will dump each element in the list which may themselves be a list or dict, so it will dump the elements those first and so on and so forth. If the elements are nested too deep(lists or dicts, which contain lists or dict, when contain lists or dicts ad infinitum) then at some point you will hit the maximum recursion depth.

The problem may be elsewhere, but without any further details that's the only thing that could be causing(if indeed that's where the error is thrown).

I have tried pickling a list of 1000 dicts(which are similar in structure to yours) and it worked just fine. Therefore either you're inadvertently building a structure that is too nested or that error is not getting thrown in the call to picke.dump()

Edit:

Try this function to see how nested the structure you're trying to dump is:

def nestedness(struct):
    if isinstance(struct, list):
        return max([0] + [nestedness(i) for i in struct]) + 1
    if isinstance(struct, dict):
        return max([0] + [nestedness(i) for i in struct.values()])+1
    return 1

If it fails with maximum recursion depth exceeded, keep raising it using sys.setrecursionlimit() until you get a proper answer. If your structure is just too nested see if you can find a way to make it less so by structuring it differently.

Edit: for posterity's sake, fixed nestedness so it handles empty dicts and sequences

I'm still confused though on how your structures ended up being so nested in the first place, because the code you posted doesn't look like it should generate such nested structures. You should make sure that you're not unnecessarily nesting stuff without being aware of it otherwise this solution will prove brittle and break in the future when 10000 ends up being too low. PS: if this answered your question you should mark the answer as accepted :) — entropy, Feb 23 '13 at 22:34
Me too,when I print list of 50 dicts it shows me 50 lines of this:{'url': u'http://shironet.mako.co.il/artist?type=lyrics&lang=1&prfid=1318&wrkid=3996', 'author': [], 'name': '', 'title': u'\u05de\u05d5\u05e8\u05d9\u05dd \u05d0\u05d9\u05e0\u05dd \u05de\u05e1\u05e4\u05e8\u05d9\u05dd'} — Dan can, Feb 23 '13 at 22:42
What does `nestedness()` return when called on your list? Btw, call it on `data` in the first line of `pickleIt()` so you're sure you're testing the data being pickled. — entropy, Feb 23 '13 at 22:43
It fails and gives the error :"ValueError: max() arg is an empty sequence" when adding this dictionary only to the list [{'url': u'http://shironet.mako.co.il/artist?type=lyrics&lang=1&prfid=1318&wrkid=3996', 'author': [], 'name': '', 'title': u'\u05de\u05d5\u05e8\u05d9\u05dd \u05d0\u05d9\u05e0\u05dd \u05de\u05e1\u05e4\u05e8\u05d9\u05dd'}] — Dan can, Feb 23 '13 at 23:06
It stoped to fail when i build the dicts with simple strings as values.There is something wrong with my implementation of inner class instance since I use the geter method like this 'title':song_obj.getSongTitle() Thanks,anyway for your reply — Dan can, Feb 23 '13 at 23:16
I've made an amendment to `nestedness()` so it doesn't fail when the structure contains empty lists of dicts. That was the problem. I was calling `max()` on an empty list. — entropy, Feb 24 '13 at 01:50

Python pickle error when dumping list of dictionaries

1 Answers1