np.savetxt does not appear to work for n-d arrays

Question

I'm trying to save a 4-D array using numpy.savetxt, and it does not appear to work.

In [13]: mat = np.zeros((3,3,2,2))
In [14]: mat[0][0][0][0] = 1.5e+10
In [15]: mat[0][0][0][1] = 1.6e+10
In [16]: mat[0][0][1][0] = 1.7e+10
In [17]: mat[0][0][1][1] = 1.8e+10
In [18]: np.savetxt("/tmp/save_mat", mat)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
----> 1 np.savetxt("/tmp/save_mat", mat)

python2.7/site-packages/numpy/lib/npyio.pyc in savetxt(fname, X, fmt, delimiter, newline, header, footer, comments)
 1158                     print(e)
 1159                     raise TypeError("Mismatch between array dtype ('%s') and "
-> 1160                                     "format specifier ('%s')"
 1161                                     % (str(X.dtype), format))
 1162         if len(footer) > 0:

 TypeError: Mismatch between array dtype ('float64') and format specifier ('%.18e %.18e %.18e')

I edited npyio.py and printed out the actual TypeError instead of the re-raised TypeError, and it was

float argument required, not numpy.ndarray

It works fine if I use the binary save method

In [20]: fd = open("/tmp/save_mat", "w")
In [21]: np.save(fd, mat)
In [22]: fd.close()

And there is a nonzero file created

$ ls -al /tmp/save_mat
-rw-r--r--  1  368 May 11 07:17 /tmp/save_mat

The numpy documentation does not say anything about the array dimensions, just that it is "array-like".

http://docs.scipy.org/doc/numpy/reference/generated/numpy.savetxt.html

numpy.savetxt(fname, X, fmt='%.18e', delimiter=' ', newline='\n', header='', footer='', comments='# ')[source]

Save an array to a text file. Parameters:

fname : filename or file handle If the filename ends in .gz, the file is automatically saved in compressed gzip format. loadtxt understands gzipped files transparently.

X : array_like Data to be saved to a text file.

Is anybody else seeing this? Is it expected behaviour?

Well, what format of output are you looking for? It's not immediately obvious how you would represent a 4-dimensional array in a two-dimensional text file. — ali_m, May 11 '16 at 14:46
`savetxt` creates a `csv` style file that can be loaded with `loadtxt`. It iterates through the `rows`, and writes each as a string formatted line. — hpaulj, May 11 '16 at 14:54
I'm just looking for something that I can save and then reload again later. so the binary `save` works for me for now, but I was curious since the implementation of `savetxt` appears to be consistent with the documentation. — Shankari, May 11 '16 at 17:15
Saving numeric data in a text file only makes sense if it needs to be read by a human. Text files are slower to read and write, and make less efficient use of storage space compared with binary formats. — ali_m, May 11 '16 at 21:57
It also has the advantage that it is much more portable, both in terms of reading by other programs/languages now, and being able to read the file ~ 5 years from now even if the binary `numpy` format has changed. As an aside, I am really not sure why this is turning into a text versus binary argument. It is a legitimate use case to want to save data as text. And if `savetxt` doesn't support n-d arrays, I don't think that it is unreasonable to expect that it should be documented as such - i.e. `Save an array to a text file` -> `Save a 1-D or 2-D array to a text file` — Shankari, Jul 04 '16 at 23:48

score 1 · Answer 1 · edited May 23 '17 at 12:08

Look again where you changed the error message:

for row in X:
   try:
       fh.write(asbytes(format % tuple(row) + newline))
   except TypeError:
   ...

Try this with your mat:

print('%.18e %.18e %.18e'%tuple(np.array([1,2,3]))) # working
for row in mat:
    print('%.18e %.18e %.18e'%tuple(row))   # your error

This is straight forward Python string formatting operation. It only works when the number of elements in row (after conversion to a tuple) matches the number of % specifiers in the format. And the elements have to match - in this case numbers that can be displayed with the %e.

There's no provision in savetxt to iterate over the higher dimensions of your array. You have to do that kind of iteration yourself.

Roughly:

f = open('txt.txt', 'w')
for block in Mat:
   for subblock in block:
       np.savetxt(f, block, fmt=...) # write to open file
       f.write('\n') # spacer line
   f.write('\n')  # another spacer
f.close()

A more detailed answer along the same line:

How to write a multidimensional array to a text file?

thanks for the detailed answer. I understand what the code is doing, but it is inconsistent with the documentation. So I just wondered whether this is expected behavior or not. I guess the answer is that it is expected, and the the documentation is wrong.. — Shankari, May 11 '16 at 17:17
There's nothing in the code to suggest than handling anything larger than 2d was contemplated. Most of the details in the docs are about the format, not the array. As far as I know there's no industry standard for writing 3d arrays as CSV files. — hpaulj, May 11 '16 at 18:32

np.savetxt does not appear to work for n-d arrays

1 Answers1