I am saving a numpy sparse array (densed) into a csv. The result is I have a 3GB csv. The problem is 95% of the cells are 0.0000. I used fmt='%5.4f'. How can I format and save such that the zeros are saved only as 0 and the non zero floats are saved with the '%5.4f' format ? I am sure I can get the 3GB down to 300MB if I can do this.
I am using
np.savetxt('foo.csv', arrayDense, fmt='%5.4f', delimiter = ',')
Thanks Regards
If you look at the source code of np.savetxt, you'll see that, while there is quite a bit of code to handle the arguments and the differences between Python 2 and Python 3, it is ultimately a simple python loop over the rows, in which each row is formatted and written to the file. So you won't lose any performance if you write your own. For example, here's a pared down function that writes compact zeros:
def savetxt_compact(fname, x, fmt="%.6g", delimiter=','):
with open(fname, 'w') as fh:
for row in x:
line = delimiter.join("0" if value == 0 else fmt % value for value in row)
fh.write(line + '\n')
For example:
In [70]: x
Out[70]:
array([[ 0. , 0. , 0. , 0. , 1.2345 ],
[ 0. , 9.87654321, 0. , 0. , 0. ],
[ 0. , 3.14159265, 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 0. ]])
In [71]: savetxt_compact('foo.csv', x, fmt='%.4f')
In [72]: !cat foo.csv
0,0,0,0,1.2345
0,9.8765,0,0,0
0,3.1416,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
Then, as long as you are writing your own savetxt function, you might as well make it handle sparse matrices, so you don't have to convert it to a (dense) numpy array before saving it. (I assume the sparse array is implemented using one of the sparse representations from scipy.sparse.) In the following function, the only change is from ... for value in row to ... for value in row.A[0].
def savetxt_sparse_compact(fname, x, fmt="%.6g", delimiter=','):
with open(fname, 'w') as fh:
for row in x:
line = delimiter.join("0" if value == 0 else fmt % value for value in row.A[0])
fh.write(line + '\n')
Example:
In [112]: a
Out[112]:
<6x5 sparse matrix of type '<type 'numpy.float64'>'
with 3 stored elements in Compressed Sparse Row format>
In [113]: a.A
Out[113]:
array([[ 0. , 0. , 0. , 0. , 1.2345 ],
[ 0. , 9.87654321, 0. , 0. , 0. ],
[ 0. , 3.14159265, 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 0. ]])
In [114]: savetxt_sparse_compact('foo.csv', a, fmt='%.4f')
In [115]: !cat foo.csv
0,0,0,0,1.2345
0,9.8765,0,0,0
0,3.1416,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0
Another simple option that may work given your requirements is the 'g' specifier. If you care more about significant digits and less about seeing exactly x number of digits and don't mind it switching between scientific and float, this does the trick well. For example:
np.savetxt("foo.csv", arrayDense, fmt='%5.4g', delimiter=',')
If arrayDense is this:
matrix([[ -5.54900000e-01, 0.00000000e+00, 0.00000000e+00],
[ 0.00000000e+00, 3.43560000e-08, 0.00000000e+00],
[ 0.00000000e+00, 0.00000000e+00, 3.43422000e+01]])
Your way would yield:
-0.5549,0.0000,0.0000
0.0000,0.0000,0.0000
0.0000,0.0000,34.3422
The above would yield instead:
-0.5549, 0, 0
0,3.436e-08, 0
0, 0,34.34
This way is also more flexible. Notice that using 'g' instead of 'f', you don't lose data (i.e. 3.4356e-08 instead of 0.0000). This obviously is dependent on what you set your precision to however.
It would be much better if you saved only the non-zeros entries in your sparse matrix (m in the example below), you could achieve that doing:
fname = 'row_col_data.txt'
m = m.tocoo()
a = np.vstack((m.row, m.col, m.data)).T
header = '{0}, {1}'.format(*m.shape)
np.savetxt(fname, a, header=header, fmt=('%d', '%d', '%5.4f'))
and the sparse matrix can be recomposed doing:
row, col, data = np.loadtxt(fname, skiprows=1, unpack=True)
shape = map(int, open(fname).next()[1:].split(','))
m = coo_matrix((data, (row, col)), shape=shape)
