Reading in a file, one chunk at a time

Question

I have a VERY large file formatted like this:

(mydelimiter)
line
line
(mydelimiter)
line
line
(mydelimiter)

Since the file is so large I can't read it all into memory at once. So I would like to read each chunk between "(mydelimiter)" at a time, perform some operations on it, then read in the next chunk.

This is the code I have so far:

with open(infile,'r') as f: 
    chunk = []
    for line in f:
        chunk.append(line)

Now, I'm not sure how to tell python "keep appending lines UNTIL you hit another line with '(mydelimiter)' in it", and then save the line where it stopped abd start there in the next iteration of the for loop.

Note: it's also not possible to read in a certain number of lines at a time since each chunk is variable length.

Please link to any questions you believe I have duplicated. I have searched stackoverflow thoroughly before posting this question and not found an answer. — Andrade, Feb 17 '17 at 04:43
The link is at the top of the page, below your question's title. — TigerhawkT3, Feb 17 '17 at 05:10

score 1 · Accepted Answer · answered Feb 17 '17 at 04:34

1

Aren't you perhaps over thinking this? Something as simple as the following code can do the trick for you

with open(infile,'r') as f: 
    chunk = []
    for line in f:
        if line == 'my delimiter':
             call_something(chunk)
             chunk=[]
        else :
             chunk.append(line)

answered Feb 17 '17 at 04:34

e4c5

52,766
11
101
134

Thanks for the response! Can you clarify what your call_something() function is doing here? – Andrade Feb 17 '17 at 04:42
well call_something is a function that you define that works with your chunk of data – e4c5 Feb 17 '17 at 04:56
Yes, thanks!! Just accepted your answer (apologies for forgetting this!) – Andrade Feb 28 '17 at 17:09
all the best with your project – e4c5 Feb 28 '17 at 17:33

Reading in a file, one chunk at a time

1 Answers1