Maximum recursion depth exceeded when using scipy.io.savemat()

Question

I'm trying to save a matrix using the scipy.io.savemat() function. However, I'm getting the following error:

RuntimeError: maximum recursion depth exceeded while calling a Python object

Here's the complete error I'm getting:

scipy.io.savemat('path/to/file.mat',dict__)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio.py", line 207, in savemat
    MW.put_variables(mdict)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 876, in put_variables
    self._matrix_writer.write_top(var, asbytes(name), is_global)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 626, in write_top
    self.write(arr)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 655, in write
    self.write_cells(narr)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 759, in write_cells
    self.write(el)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 655, in write
    self.write_cells(narr)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 759, in write_cells
    self.write(el)
.
.
.


File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 759, in write_cells
    self.write(el)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 655, in write
    self.write_cells(narr)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 758, in write_cells
    for el in A:
  File "C:\Python27\lib\site-packages\numpy\matrixlib\defmatrix.py", line 316, in __getitem__
    out = N.ndarray.__getitem__(self, index)
  File "C:\Python27\lib\site-packages\numpy\matrixlib\defmatrix.py", line 292, in __array_finalize__
    if (isinstance(obj, matrix) and obj._getitem): return
RuntimeError: maximum recursion depth exceeded while calling a Python object

I tried to increase the recursion limit using sys.setrecursionlimit(10000) but it didn't change anything. I also tried the resource.setrlimit() fix but it didn't help neither.

Does anybody have an idea what's the problem here or why is this happening? Is there any way to fix this error? Thanks in advance for your help!

P.S: I'm getting this error on Windows and Linux!

Elie

My _guess_ is that you have a circular reference in `dict__`. It's hard to be more specific without seeing your code. Please post a [mcve] that we can run to reproduce this error. — PM 2Ring, Aug 13 '16 at 15:56
@PM2Ring sorry, but I don't understand what do you mean by circular reference? The dictionary contains variables and arrays created by the program. It was all working fine until I added 2 arrays to this dictionary (195x195). Thanks for your response! — Elie khalifeh, Aug 13 '16 at 16:02
I can't post an example of the code since it is confidential and I couldn't reproduce this error with another script. I appreciate your help! — Elie khalifeh, Aug 13 '16 at 16:04
A circular reference is when you have something like collection A contains collection B as an element, but collection B also contains collection A as an element. Or you could have a longer chain like A contains B contains C contains A. Etc. — PM 2Ring, Aug 13 '16 at 16:06
We don't need to see your actual code: it's probably too large for SO anyway. But if you can't show us _something_ that gives rise to the same bug it will be very hard for us to debug this problem. — PM 2Ring, Aug 13 '16 at 16:08
@PM2Ring No, I don't think there's a circular reference in dict__. — Elie khalifeh, Aug 13 '16 at 16:11
@PM2Ring You're right, normally I should post a sample of the code, but the problem is that I can't reproduce this error with another script. I tried almost everything! — Elie khalifeh, Aug 13 '16 at 16:12
You could always [use a debugger](https://docs.python.org/2.7/library/pdb.html). Place `import pdb; pdb.set_trace()` right before the call to `savemat`. Then step through the code. Set a break point on `C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py` line 655 and investigate why `self.write_cells(narr)` is getting called (presumably) ad infinitum. Use, for example, `p narr` to print the value of `narr`. Check if those `narr`s match your expectation. — unutbu, Aug 13 '16 at 16:17
I can't access mio5.py to set a breakpoint on line 655. Is there a specific way to do it? I'm trying to append the path and then import mio5 but I'm getting the error "ValueError: Attempted relative import in non-package" @unutbu — Elie khalifeh, Aug 13 '16 at 17:02
Okay so after going through the debug, I found that after adding those 2 matrices, some of the other variables are being converted to objects (and that's something I can't control), so I'm wondering if this could be the reason why savemat is producing the error above? @unutbu thanks all for your time and help! — Elie khalifeh, Aug 13 '16 at 18:21
I think you might be on to something. Have you been able to identify what object `arr` is causing the infinite loop? — unutbu, Aug 13 '16 at 18:32
Yes, when the variable's dtype = object, the error in question is raised! @unutbu — Elie khalifeh, Aug 13 '16 at 18:34
Can you construct a simple example that reproduces the problem? One way to reproduce a maximum recursion error is if `arr` is a recursive array such as `arr = np.random.random((5,4)).astype('O'); arr[0,0] = arr`. — unutbu, Aug 13 '16 at 18:43
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/120877/discussion-between-elie-khalifeh-and-unutbu). — Elie khalifeh, Aug 13 '16 at 18:45

Elie khalifeh · Accepted Answer · 2016-08-13T20:24:39.347

1

Okay so just for the record, the cause of the problem is two variables in dict__ that were being converted to object (normally they're supposed to be float64). When I was trying to save dict__ in file.mat using scipy.io.savemat(), the error above was occurring when the variable's dtype is object. I don't know though:

Why savemat() was raising this specific error when the variable that's being saved is an object
Why python is converting, by itself, a float64 array to an object

Thanks everyone for your help especially @unutbu for the debugging tip!

edited Aug 13 '16 at 20:24

answered Aug 13 '16 at 19:46

Elie khalifeh

108
6

If a variable is a list, `savemat` turns it into an array before saving (`np.array(...)`). If the elements of the list differ in size, the resulting array is `dtype object`. `savemat` saves object arrays as MATLAB cell arrays. Hence error in `write_cells`. See: http://stackoverflow.com/questions/38960464/using-scipy-io-savemat-to-save-nested-lists – hpaulj Aug 16 '16 at 05:13
@hpaulj The variables I'm sending to savemat through dict__ are arrays only, no lists. – Elie khalifeh Aug 16 '16 at 18:51

Maximum recursion depth exceeded when using scipy.io.savemat()

1 Answers1