Reading file with N lines at once in python

Question

I am reading xyz trajectory files a lot. These files are structured in a way, that information corresponding to a time frame is stored in N lines.

I would like to write an iterator similar to:

file=open(...)
for line in file:
   analyze(line)

but reading N line at once:

file=Myopen(...,N=n)
for Nlines in file:
    analyze(Nlines)

Since the files are huge, I do not want to read the whole into memory, but the purpose is not to gain efficiency but to make a clean and reuseable code. Of course, one could check the index%N==0, and analyze when it is true, but I am a bit sick of writing that few lines over, and over, and over....

Comments and answers are more than appreciated!

you might want to look at: http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks — Ozgur Vatansever, May 11 '17 at 16:26

score 1 · Accepted Answer · answered May 12 '17 at 08:05

The itertools documentation has a recipe for a generator function that does what you want:

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

If you don't need to handle files that aren't an exact multiple of three lines long, you can simplify things a bit and just use for nlines in zip(*[file]*5) directly in your code.

score 0 · Answer 2 · answered May 12 '17 at 07:52

0

For instance:

file=Myopen(...,N=n)
Nlines = []
for i in range(N):
     Nlines.append(file.readline)
analyze( ''.join(Nlines) )

answered May 12 '17 at 07:52

stovfl

14,998
7
24
51

Reading file with N lines at once in python

2 Answers2