Is there a python library for line based file reading?

Question

Possible Duplicate:
Python: How to read huge text file into memory

To process a large text file(1G+) line by line , random access by any line number is desired, most importantly, without loading the whole file content into RAM. Is there a python library to do that?

It is beneficial when analyzing a large log file, read only is enough.

If there is no such standard library, I have to seek an alternative method: Find a set of function/class that can return the N-th line of sub-string from a big string-like object, so that I can mmap(yes, I mean memory-mapped file object) the file to that object then do line-based processing.

Thank you.

PS: A log file is almost sure to have variable line length.

Are all lines of the same length? – Tim Pietzcker Feb 08 '12 at 07:09 — Tim Pietzcker, Feb 08 '12 at 07:09
can you preprocess the file to create an index of sorts? – anijhaw Feb 08 '12 at 07:14 — anijhaw, Feb 08 '12 at 07:14

Kimvais · Accepted Answer · 2012-02-08T07:45:35.363

1

I think that something like below might work, since the file object's method readline() reads one line at a time. If the lines are of arbitrary length, you need to index the positions like follows.

lines = [0]
with open("testmat.txt") as f:
    while f.readline():
        lines.append(f.tell())
    # now you can read an arbitrary line:
    f.seek(lines[1235])
    line = f.readline()

If the lines were of same length, you could just do f.seek(linenumber*linelenght)

edited Feb 08 '12 at 07:45

answered Feb 08 '12 at 07:31

Kimvais

38,306
16
108
142

1

This looks like the "index of sorts" to which @anijhaw referred. – Jim DeLaHunt Feb 08 '12 at 07:35

Is there a python library for line based file reading?

1 Answers1