How to save a large array in txt file in python

Question

I have 2D array:

import numpy as np

output = np.array([1,1,6])*np.arange(6)[:,None]+1

output
Out[32]: 
array([[ 1,  1,  1],
       [ 2,  2,  7],
       [ 3,  3, 13],
       [ 4,  4, 19],
       [ 5,  5, 25],
       [ 6,  6, 31]])

I tried to use np.savetxt('file1.txt', output, fmt='%10d') i have got the result in one line only

How can I save it in txt file simillar to :

3 separate columns, each column has name (x,y,z)

Please note: the original array too large (40000000 rows and 3 columns), I am using Python 3.6 I have tried the solutions in here and here but, it does not work with me

40mio*3*3 characters for the data and spacing are roughly 350mb assuming 1 byter per character - thats not a too big file. — Patrick Artner, Dec 31 '17 at 21:38
@PatrickArtner: but if we use text, it uses separators (tab and new line), and some data (e.g. floats) can use ~10-20 digits. — Willem Van Onsem, Dec 31 '17 at 21:39
@WillemVanOnsem sure, his demodata are ints in range below 99 though and I rounded up ;) just trying to get a feel about his problem — Patrick Artner, Dec 31 '17 at 21:40
@PatrickArtner: in case it is 0-99 uniformly, it is ~ 1.044 GiB. — Willem Van Onsem, Dec 31 '17 at 21:43
"it does not work with me" **how** does it not work? I didn't downvote, but please read [this](http://idownvotedbecau.se/itsnotworking/) — juanpa.arrivillaga, Dec 31 '17 at 21:45
I can save the array in txt file, but, it appears in one line not organized in rows and columns similar above — Noor, Dec 31 '17 at 21:51
You should tell `np.savetxt(...)` to use windows-style-newlines, by default it uses unix-style-ones and your editor is probably not smart enough to show those correctly as newlines. See answer. — Patrick Artner, Dec 31 '17 at 22:15
@WillemVanOnsem I overlooked the %10d and used 3chars optimistically. You are far closer to the filesize - thanks for making me think — Patrick Artner, Dec 31 '17 at 22:17

Patrick Artner · Accepted Answer · 2017-12-31T22:20:31.750

Noor, let me guess - you are using windows notepad to view the file?

I use Notepad++ which is smart enough to understand Unix-style-Lineendings which are used (by default) when creating files by np.savetxt() even when operated under windows.

You might want to explicitly specify newline="\r\n" when calling savetxt.

np.savetxt('file1.txt', output, fmt='%10d' ,header= "       x          y          z", newline="\r\n")

Doku: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.savetxt.html

I am not sure about your data, but this:

import numpy as np

output = np.array([1,1,6])*np.arange(60)[:,None]+1

print(output)


np.savetxt('file1.txt', output, fmt='%10d' ,header= "       x          y          z")

Produces this output:

#        x          y          z
         1          1          1
         2          2          7
         3          3         13 
       === snipped a few lines ===
        58         58        343
        59         59        349
        60         60        355

for me.

for np.arange(1000000) its about 32MB big and similarly formatted...
for np.arange(10000000) its about 322MB big and similarly formatted...

willem-van-onsem 1+Gb was far closer.

I did not account for the spacing of fixed 10 chars per number, my bad.

Thanks, Patrick Artner, you are right, Notepad++, and EmEditor, both of them display the array in columns and rows — Noor, Dec 31 '17 at 22:39
@Noor - specify the newline and windows style one are used- wich adds 40.000.000 bytes to your overall file size ;) — Patrick Artner, Dec 31 '17 at 22:41

How to save a large array in txt file in python

1 Answers1