-2

I would like to know can we read line by line in an array. For example:

array([[ 0.28,  0.22,  0.23,  0.27],
       [ 0.12,  0.29,  0.34,  0.21],
       [ 0.44,  0.56,  0.51,  0.65]])

To read the first line as in array form to perform some operation and then continue with second row array:

array([0.28,0.22,0.23,0.27])

What produces the above array is because of these two lines of code:

from numpy import genfromtxt
single=genfromtxt('single.csv',delimiter=',')

single.csv

0.28,  0.22,  0.23,  0.27
0.12,  0.29,  0.34,  0.21
0.44,  0.56,  0.51,  0.65

Using readlines() seem like producing list instead of array. In my case, I'm using csv file. I'm trying to use the rows of values line by line instead of using them all together to avoid memory error. Can anyone help me?

with open('single.csv') as single:
    single=single.readlines()
maazza
  • 7,016
  • 15
  • 63
  • 96
Xiong89
  • 767
  • 2
  • 13
  • 24

3 Answers3

10

you can use np.fromstring 

import numpy as np
with open('single.csv') as f:
    lines=f.readlines()
    for line in lines:
        myarray = np.fromstring(line, dtype=float, sep=',')
        print(myarray)

see http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.fromstring.html and How to read csv into record array in numpy?

Community
  • 1
  • 1
maazza
  • 7,016
  • 15
  • 63
  • 96
  • Sorry I'm not familiar with np.fromstring. And it returned this error: NameError: name 'double' is not defined – Xiong89 Feb 23 '16 at 13:40
8

Seems you don't have experience reading files in Python. Let me work through an example in some detail, in an Ipython iteractive session

Create a multiline text to simulate your file

In [23]: txt="""0.28,  0.22,  0.23,  0.27
0.12,  0.29,  0.34,  0.21
0.44,  0.56,  0.51,  0.65"""

split it into lines to simulate the result of readlines

In [24]: txt=txt.splitlines(True)

In [25]: txt
Out[25]: 
['0.28,  0.22,  0.23,  0.27\n',
 '0.12,  0.29,  0.34,  0.21\n',
 '0.44,  0.56,  0.51,  0.65']

I can transform it into an array with genfromtxt (you could have passed the result to readlines to genfromtxt like this.

In [26]: np.genfromtxt(txt, delimiter=',')
Out[26]: 
array([[ 0.28,  0.22,  0.23,  0.27],
       [ 0.12,  0.29,  0.34,  0.21],
       [ 0.44,  0.56,  0.51,  0.65]])

I can iterate over the lines, strip off the \n and split on ','

In [27]: for line in txt:
    print line.strip().split(',')
   ....:     
['0.28', '  0.22', '  0.23', '  0.27']
['0.12', '  0.29', '  0.34', '  0.21']
['0.44', '  0.56', '  0.51', '  0.65']

I can transform each string into a float with a list comprehension:

In [28]: for line in txt:                                  
    print [float(x) for x in line.strip().split(',')]
   ....:     
[0.28, 0.22, 0.23, 0.27]
[0.12, 0.29, 0.34, 0.21]
[0.44, 0.56, 0.51, 0.65]

Or by putting the iteration in another list comprehension, I can get a list of lists of numbers:

In [29]: data=[[float(x) for x in line.strip().split(',')] for line in  txt]

In [30]: data
Out[30]: [[0.28, 0.22, 0.23, 0.27], [0.12, 0.29, 0.34, 0.21], [0.44, 0.56, 0.51, 0.65]]

I can turn that into an array

In [31]: np.array(data)
Out[31]: 
array([[ 0.28,  0.22,  0.23,  0.27],
       [ 0.12,  0.29,  0.34,  0.21],
       [ 0.44,  0.56,  0.51,  0.65]])

genfromtxt is essentially going through that sequence - reading the lines,spliting them, converting strings to values, and finally creating an array from the list.

There are short cuts, but I think you would benefit from working through these steps in detail. It's as much an exercise in basic Python string and list manipulation as it is about arrays.

hpaulj
  • 221,503
  • 14
  • 230
  • 353
0
for list in array:
    print(list)
    for item in list:
        print(item)

gives the output:

[0.28, 0.22, 0.23, 0.27]
0.28
0.22
0.23
0.27
[0.12, 0.29, 0.34, 0.21]
0.12
0.29
0.34
0.21
[0.44, 0.56, 0.51, 0.65]
0.44
0.56
0.51
0.65
Fauxpas
  • 86
  • 6