How to read a 100GB one-line text file in Python?

Question

I'm on Windows platform and using Python 3. Since the default behavior of file readers is to consume file line by line, I have difficulty dealing with my 100GB text file which has only one line.

I'm aware of solutions such as this for introducing a custom record separator for replacing a frequent character with \n; but I wonder is there anyway that I could consume and process my file only via Python?

I have only 8GB of ram. My file is the records of sales (including item, price, buyer, ...). My processing of the file is mostly editing price numbers. Records are separated from each other using | character.

you cannot read 100GB in memory if you have only 8GB RAM. You should look for an iterative solution — pythonic833, Aug 16 '19 at 12:40
@C.Nivs Agreed, but OP should really tell us what he wants to do with the file's contents... — ruohola, Aug 16 '19 at 12:40
Iteratively reading is no solution to that. A well formulated question would ask how to process the amount of data and what kind of process is wanted. — pythonic833, Aug 16 '19 at 12:41
Possible duplicate of [Lazy Method for Reading Big File in Python?](https://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python) — mkrieger1, Aug 16 '19 at 12:42
Why not keep on reading 1024 bytes until `|` is found, then process the record? — Sniper, Aug 16 '19 at 12:56
https://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python/30775393#30775393 — Jainil Patel, Aug 16 '19 at 20:30

score 5 · Answer 1 · answered Aug 16 '19 at 12:41

# !/usr/bin/python3
import os, sys

# Open a file
fd = os.open("foo.txt",os.O_RDWR)

# Reading text
ret = os.read(fd,12)
print (ret.decode())

# Close opened file
os.close(fd)
print ("Closed the file successfully!!")

or

with open(filename, 'rb') as f:
    while True:
        buf = f.read(max_size)
        if not buf:
            break
        process(buf)

or

from functools import partial

with open('somefile', 'rb') as openfileobject:
    for chunk in iter(partial(openfileobject.read, 1024), b''):
        do_something()

Second solution worked like a charm. This should be the accepted answer. — Teasel, Sep 24 '21 at 06:41

score 2 · Accepted Answer · answered Aug 16 '19 at 14:20

if you're running a 64bit OS, you could mmap the whole file in and let your OS actually do the reading in the background for you. mmaped files mostly present the same interface as a bytearray, so you could do things like:

import mmap

with open('largefile.txt', 'rb') as fd:
    buf = mmap.mmap(fd.fileno(), 0, access=mmap.ACCESS_READ)

you'd then be able to use buf as a normal bytearray, with operations like this to iterate over your seperator:

def split_sep(buf, sep=b'|'):
  pos = 0
  while True:
    end = buf.find(sep, pos)
    if end == -1:
       break
    yield buf[pos:end]
    pos = end + 1
  yield buf[pos:]

but this is just a demo. you'd probably want to do something more complicated, maybe decoding from bytes before yielding etc.

How to read a 100GB one-line text file in Python?

2 Answers2