14

Suppose I have a text file that looks like this:

33 3
46 12
23 10 23 11 23 12 23 13 23 14 23 15 23 16 24 10 24 11 24 12 24 13 24 14 24 15 24 16 25 14 25 15 25 16 26 16 27 16 28 16 29 16
33 17 33 18 33 19 34 17 34 18 34 19 35 17 35 18 35 19 36 19
41 32 41 33 42 32 42 33

I would like to read each line into a separate array of integers, as in (pseudo code):

for line in textfile:  
    currentArray = firstLine  
    do stuff with currentArray

where in the first iteration, currentArray would be

array([33, 3])

and in the second iteration, currentArray would be

array([46, 12])

until the last iteration, when currentArray would be

array([41, 32, 41, 33, 42, 32, 42, 33])

Basically, I would like to have the functionality of the numpy function loadtxt:

currentArray = loadtxt('scienceVertices.txt', usecols=() )

Except instead of usecols, being able to specify the row, e.g.,

currentArray = loadtxt('scienceVertices.txt', userows=(line) )

ChristopheD
  • 112,638
  • 29
  • 165
  • 179
lookitsmarc
  • 227
  • 2
  • 3
  • 8
  • Depending on the size of your data file, it may be significantly more efficient not to load all the lines into memory at once. This is possible, since iterating over a `file` object in Python will give you the lines one at a time, loaded sequentially from disc with IO buffering. OTOH, if you only have a few megabytes of data it may be easier not to bother. – Katriel Feb 11 '11 at 21:59
  • @katrielalex In my application I will likely only be reading kilobytes of data, but I'll keep your suggestion in mind if I move onto larger files. Thanks. – lookitsmarc Feb 11 '11 at 22:59

4 Answers4

17

Here's a one-liner:

arrays = [np.array(map(int, line.split())) for line in open('scienceVertices.txt')]

arrays is a list of numpy arrays.

Paul
  • 42,322
  • 15
  • 106
  • 123
  • Well played. I didn't ask for it to be in a list of numpy arrays but I think this is better for my application. Thank you. – lookitsmarc Feb 11 '11 at 21:55
  • This solution, at least in python3, return a list of maps. How can I use something similar to retrieve numpy arrays? – heracho Apr 19 '22 at 16:39
  • @heracho Just wrap the map(...) function with a list() function. – Paul Apr 20 '22 at 01:52
6
for line in textfile:
  a = np.array([int(v) for v in line.strip().split(" ")])
  # Work on your array
payne
  • 13,833
  • 5
  • 42
  • 49
3

You can also use numpy.fromstring()

for line in f:
    a = numpy.fromstring(line.strip(), dtype=int, sep=" ")

or -- if you want full flexibility -- even numpy.loadtxt():

for line in f:
    a = numpy.loadtxt(StringIO.StringIO(line), dtype=int)

For long lines, these solution will perform better than the Python code in the other answers.

Sven Marnach
  • 574,206
  • 118
  • 941
  • 841
2
f = open("file", "r")
array = []
line = f.readline()
index = 0
while line:
    line = line.strip("\n")
    line = line.split()
    array.append([])
    for item in line:
        array[index].append(int(item))
    line = f.readline()
    index += 1
f.close()

print array
Santa
  • 11,381
  • 8
  • 51
  • 64
Asterisk
  • 3,534
  • 2
  • 34
  • 53