2

I got several numpy arrays (X, ) from a loop and would like to store them. I know that I could append them to a conventional python list

A = []
for i in range(10):
  A.append(some_numpy_array)

However, this solution does not look elegant and cannot be parsed to numpy.savetxt. 

numpy.savetxt("out.txt", A)

Is there any other solution?

Kevin
  • 45
  • 1
  • 2
  • 5

1 Answers1

1

So you have a list of arrays

In [6]: alist = [np.arange(i) for i in range(3,7)]
In [7]: alist
Out[7]: 
[array([0, 1, 2]),
 array([0, 1, 2, 3]),
 array([0, 1, 2, 3, 4]),
 array([0, 1, 2, 3, 4, 5])]

What's inelegant about that? You could wrap it in an object array

In [8]: arr=np.array(alist)
In [9]: arr
Out[9]: 
array([array([0, 1, 2]), array([0, 1, 2, 3]), array([0, 1, 2, 3, 4]),
       array([0, 1, 2, 3, 4, 5])], dtype=object)

That loses list methods like append, but gains array ones like reshape. And array math has to resort to the python level iteration, the speed equivalent of list comprehensions. And there's one big gotcha - It all the sublists have the same length, the result is a 2d int array, not an object one.

savetxt, given a list, will turn it into an array as I did, and then try to write 'rows'. savetxt is designed to produced CSV output - neat rows and columns with delimiter. That display doesn't make sense with this list, does it?

In [11]: np.savetxt('test.txt',alist,fmt='%s',delimiter=',')
In [12]: cat test.txt
[0 1 2]
[0 1 2 3]
[0 1 2 3 4]
[0 1 2 3 4 5]

This wrote the list/array as a 1d array with general string formatting for each line. Effectively

In [13]: for row in alist:
    ...:     print(row)
    ...:     
[0 1 2]
[0 1 2 3]
[0 1 2 3 4]
[0 1 2 3 4 5]

You could do repeated savetxt on an open file, writing one array at a time:

In [18]: with open('test.txt','wb') as f:
    ...:     for row in alist:
    ...:         np.savetxt(f, [row], fmt='%5d',delimiter=',')      
In [19]: cat test.txt
    0,    1,    2
    0,    1,    2,    3
    0,    1,    2,    3,    4
    0,    1,    2,    3,    4,    5

People run into the same issue when trying to write 3d arrays, or other arrays that don't fit the simple 2d numeric array model. You don't have to use savetxt to write text to a file. Plain Python has sufficient tools for that.

 

Cabbage soup
  • 1,344
  • 1
  • 18
  • 26
hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • Very clear reply! One follow-up question, np.savetxt the list-of-array seems to format badly if the array is too long. With the row-by-row method, how can I add some string at the beginning of each line? – Kevin Apr 16 '17 at 16:37
  • `%s` uses the object's `str()` method, also used by `print`. For large arrays that produces multiple lines and ellipsis. To some degree that's controlled with `printoptions`, but ultimately it's designed to give a pretty view of the array, not a complete 'machine-readable' text representation. Converting the individual arrays `tolist()` might give more satisfactory output. – hpaulj Apr 16 '17 at 16:54
  • How do you intend to use or read this text file? Most `csv` readers expect regular columns. – hpaulj Apr 16 '17 at 16:59