Python Memory Error when reading large files , need ideas to apply mutiprocessing in below case?

Question

I have the file which stores the data in the below format

TIME[04.26_12:30:30:853664]ID[ROLL:201987623]MARKS[PHY:100|MATH:200|CHEM:400]
TIME[03.27_12:29:30.553669]ID[ROLL:201987623]MARKS[PHY:100|MATH:1200|CHEM:900]
TIME[03.26_12:28:30.753664]ID[ROLL:2341987623]MARKS[PHY:100|MATH:200|CHEM:400]
TIME[03.26_12:29:30.853664]ID[ROLL:201978623]MARKS[PHY:0|MATH:0|CHEM:40]
TIME[04.27_12:29:30.553664]ID[ROLL:2034287623]MARKS[PHY:100|MATH:200|CHEM:400]

Below method I found to fulfill the need given in this question please refer this link for clarification

import re
from itertools import groupby

regex = re.compile(r"^.*TIME\[([^]]+)\]ID\[ROLL:([^]]+)\].+$")
def func1(arg) -> bool:
    return regex.match(arg)


def func2(arg) -> str:
    match = regex.match(arg)
    if match:
        return match.group(1)
    return ""


def func3(arg) -> int:
    match = regex.match(arg)
    if match:
        return int(match.group(2))
    return 0
with open(your_input_file) as fr:
    collection = filter(func1, fr)
    collection = sorted(collection, key=func2)
    collection = sorted(collection, key=func3)
    for key, group in groupby(collection, key=func3):
        with open(f"ROLL_{key}", mode="w") as fw:
            fw.writelines(group)

The above function is creating the files according to my wish also , it's sorting the file_contents according to time stamps and I am getting correct output so i tried it for large files of the size 1.7 GB it's giving memory error I tried to use the following method

Failed attempt:

  with open(my_file.txt) as fr:
        part_read = partial(fr.read, 1024 * 1024)
        iterator = iter(part_read, b'')
        for index, fra in enumerate(iterator, start=1):
         collection = filter(func1, fra)
         collection = sorted(collection, key=func2)
         collection = sorted(collection, key=func3)
         for key, group in groupby(collection, key=func3):
            fw=open(f'ROLL_{key}.txt','a')
            fw.writelines(group)

This attempt doesn't gave me any results means there was no file created at all it's taking unexpectedly huge time , i found in many of the answers to read file line by line then how I will then sort it , please suggest me suggestions to improve this code or any new idea if I need to use multiprocessing here to process faster ,if that is the case How to use it?

And One main condition with me is I can't store it any data structure since file can be huge in size

To sort the file in memory, you need to fit the file into memory. Normally I would use a database for this problem. If that is out of reach, then I would use an external sort utility like `sort` available on Linux that handles files too large to fit into memory. — Mike Organek, Jul 03 '20 at 15:17
How you would do it can you give example please like how can i use it for my case? — Ankur, Jul 03 '20 at 15:26

score 1 · Answer 1 · answered Jul 03 '20 at 07:56

And if you want read file by chunk, use this:

import re
from functools import partial
from itertools import groupby
from typing import Tuple

regex = re.compile(r"^.*TIME\[([^]]+)\]ID\[ROLL:([^]]+)\].+$")
def func1(arg) -> bool:
    return regex.match(arg)


def func2(arg) -> Tuple[str, int]:
    match = regex.match(arg)
    if match:
        return match.group(1), int(match.group(2))
    return "", 0

def func3(arg) -> int:
    match = regex.match(arg)
    if match:
        return int(match.group(2))
    return 0

def read_in_chunks(file_object, chunk_size=1024*1024):
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data

with open('b.txt') as fr:
    for chunk in read_in_chunks(fr):
        collection = filter(func1, chunk.splitlines())
        collection = sorted(collection, key=func2)
        for key, group in groupby(collection, key=func3):
            with open(f"ROLL_{key}", mode="wa") as fw:
                fw.writelines(group)

Hi thank you for answering my question i want to change the line your method is writing in the same line on the newly made file — Ankur, Jul 03 '20 at 09:22

score 0 · Answer 2 · answered Jul 03 '20 at 07:45

let's try to sort at one time.

import re
from itertools import groupby
from typing import Tuple

regex = re.compile(r"^.*TIME\[([^]]+)\]ID\[ROLL:([^]]+)\].+$")
def func1(arg) -> bool:
    return regex.match(arg)


def func2(arg) -> Tuple[str, int]:
    match = regex.match(arg)
    if match:
        return match.group(1), int(match.group(2))
    return "", 0

def func3(arg) -> int:
    match = regex.match(arg)
    if match:
        return int(match.group(2))
    return 0

with open('b.txt') as fr:
    collection = filter(func1, fr)
    collection = sorted(collection, key=func2)
    for key, group in groupby(collection, key=func3):
        with open(f"ROLL_{key}", mode="w") as fw:
            fw.writelines(group)

Python Memory Error when reading large files , need ideas to apply mutiprocessing in below case?

2 Answers2

Linked