6

I'm trying to pickle instance of my cellular automata class, but I get this error:

RuntimeError: maximum recursion depth exceeded while calling a Python object

My cellular automata consist from list of cells (and bunch of other things) where each cell has pointer to it's neighbours. In this particular CA, there is 256 cells. Now, I know that pickler should be able to recognise already pickled objects.

From docs:
*The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again.

So I don't really know, why I exceeding max recursion depth.

I think that maybe pickler does depth-first pickling, so that it first follow pointers, exceed recursion stack and then raise exception. I know I can extend maximum recursion depth with sys.setrecursionlimit(), but I don't consider that good nor scalable solution.

First question: Does pickler depth-first pickling?
Second question: Any idea how to prevent this exception?

sjudǝʊ
  • 1,565
  • 1
  • 23
  • 34
  • any code that you can post? – Aswin Murugesh Apr 13 '13 at 19:57
  • Yes, `pickle` goes depth-first. Unfortunately I don't think there is a around this. Try `cPickle` but it'll probably give the same error. – Elmar Peise Apr 13 '13 at 20:08
  • @AswinMurugesh actually I have too much code to post. However here are some key files if you are interested: [neighbourhood](https://github.com/mirobeka/cellular-automata/blob/master/cellular_automata/lattices/neighbourhoods.py) [cell](https://github.com/mirobeka/cellular-automata/blob/master/cellular_automata/cells/regular.py#L21) [pickling](https://github.com/mirobeka/cellular-automata/blob/master/cellular_automata/lattices/equiangular.py#L210) Exp: cPickle is the same, I tried it before. I was afraid of this answer – sjudǝʊ Apr 13 '13 at 20:45
  • @ExP do you have some more specific info / resources about how pickle works internally? Maybe creating subclass of pickler? There's always way. I just need to take a closer look :) – sjudǝʊ Apr 15 '13 at 21:33
  • @sjudǝʊ If you look at the output of `pickle` given dummy data, you will see that it gos depth first. Since I guess that the format does not allow forward references, I'm afraid you won't be able to modify or create subclass of it in order to solve your problem... – Elmar Peise Apr 16 '13 at 08:32
  • Possible duplicate of [Hitting Maximum Recursion Depth Using Python's Pickle / cPickle](http://stackoverflow.com/questions/2134706/hitting-maximum-recursion-depth-using-pythons-pickle-cpickle) – Ciro Santilli OurBigBook.com Jan 16 '17 at 14:29

1 Answers1

6

So, as @ExP said, pickler does depth-first pickling which causes recursion exceeded exception. Anyway, I found solution to this issue here bugs.python.org. That means for python 3.1 pickler works even on recursive data such as graphs for example.

There is also little less elegant solution which takes a lot more time to pickle some recursive data, but it's simple (just a matter of few lines of code). Link here.

As it seems, it's probably time to start slowly moving towards the python3. Hope that someone find this answer usefull.

sjudǝʊ
  • 1,565
  • 1
  • 23
  • 34
  • If you end up using the fix from http://bugs.python.org/issue2480, make sure you use the python file nonrecursivepickler-fixed.py over the other one. Also of note, this pickle solution compresses much less than vanilla pickle, so your pickle files will be larger. Thanks for the solution! – kevin948 Jun 24 '14 at 15:20
  • 2
    Although this issue2480 version works, it is very inefficient for large structures because of its use of list.pop(0) and list.extend. On my large data structure it was taking 10 hours to pickle. A rewrite to use deques reduced that to 4 minutes. – strubbly Jul 23 '15 at 20:01