Process large file in chunks

Question

I have a large file which has two numbers per line and is sorted by the second column. I make a dictionary of lists keyed on the first column.

My code looks like

from collections import defaultdict
d = defaultdict(list)
for line in fin.readline():
    vals = line.split()
    d[vals[0]].append(vals[1])
process(d)

However the input file large is too large so d will not fit into memory.

To get round this I can in principle read in chunks of the file at a time but I need to make an overlap between the chunks so that process(d) won't miss anything.

In pseudocode I could do the following.

1. Read 100 lines creating the dictionary d.
2. Process the dictionary d
3. Delete everything from d that is not within 10 of the max value seen so far.
4. Repeat but making sure we don't have more than 100 lines worth of data in d at any time.

Is there a nice way to do this in python?

Update. More details of the problem. I will use d when reading in a second file of pairs where I will output the pair if depending on how many values there are in the list associated with the first value in d which are within 10. The second file is also sorted by the second column.

Fake data. Let's say we can fit 5 lines of data into memory and we need the overlap in values to be 5 as well.

1 1
2 1
1 6
7 6
1 16

So now d is {1:[1,6,16],2:[1],7:[6]}.

For the next chunk we only need to keep the last value (as 16-6 > 5). So we would set

d to be {1:[16]} and continue reading the next 4 lines.

Answer 1

Have you tried out the Pandas library, and in particular reading your data into a DataFrame then using groupby on the first column?

Pandas will let you do a lot of bulk operations effectively across your data, and you can read it in lazily if you want to.

Answer 2

You don't need default dict unless something strange is going on with the file, but you haven't mentioned what that is. Instead, use a list, which keeps your data in line order, that way you can process it using the appropriate slices thus:

d = []
for line in fin.readline():
    vals = line.split()
    d.append(vals)
    if not len(d)%100:
        process(d)
        d = d[90:]
process(d)

Answer 3

You could do this something like this:

n_process = 100
n_overlap = 10
data_chunk = []
for line in fin.readline():
    vals = line.split()
    data_chunk.append(vals)
    if len(data_chunk) == n_process:
        process(data_chunk)
        data_chunk = data_chunk[-n_overlap:]

When using a dictionary, data can be overwritten if multiple occurrences of numbers in the first column are present in a data sample. Also notice, that you need to use OrderedDict, since a dict does not have an order in python. However, in my opinion, OrderedDict is in most cases a sign of bad code design.

And by the way: we still don't know why you're trying to do this way…