Is there a way in Python to read text file token-by-token?

Question

I need to process a BIG text file that contains space-separated float numbers in ASCII representation:

1.0012 0.63 18.201 -0.7911 92.2869 ...

How do I read these numbers one-by-one (not entire file and not line-by-line) using built-in Python tools? As sample, the C source code to solve this task looks like:

float number;
FILE *f = fopen ("bigfile.txt", "rt");
while (!feof (f)) {
    fscanf (f, "%f", &number);
    /* ... processing the number here ... */
}
fclose (f);

Does this answer your question? [Reading a file with a specified delimiter for newline](https://stackoverflow.com/questions/16260061/reading-a-file-with-a-specified-delimiter-for-newline) — tevemadar, Jun 18 '21 at 10:35
also https://stackoverflow.com/questions/35459765/read-file-up-to-a-character and https://stackoverflow.com/questions/10183784/is-there-a-way-to-read-a-file-in-a-loop-in-python-using-a-separator-other-than-n — tevemadar, Jun 18 '21 at 10:37
Links you provide are acceptable workarounds, but I'm wondering that standard Python file object doesn't have a method to do that. Thank you. — R0bur, Jun 18 '21 at 13:54

score 2 · Answer 1 · answered Jun 18 '21 at 10:35

You can try read file char by char, specifying the Chunk size to 1, and then recognize if a word is complete.

with open('file', 'r') as openedFile:
    for chunk in iter(partial(openedFile.read, 1), b''):
        ...

Links usefulls:

https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects

score 1 · Answer 2 · answered Jun 18 '21 at 10:22

1

You should be able to just read one line at a time and then split() each line to get the number tokens:

with open('file.txt') as f:
    lines = f.readlines()

for line in lines:
    tokens = line.split()
    for token in tokens:
        # process number here

answered Jun 18 '21 at 10:22

Tim Biegeleisen

502,043
27
286
360

1

But the problem is, the OP specified space separated values, with no mention of newlines. – Red Jun 18 '21 at 13:10
@AnnZen My answer should still be valid for a single line file. – Tim Biegeleisen Jun 18 '21 at 13:13
1

I know, but it then becomes "read in the entire line and split it afterwards", whereas the OP wants to read in the line gradually. – Red Jun 18 '21 at 13:14

Andrew Eckart · Answer 3 · 2021-06-18T10:31:36.013

If a line-by-line solution is not viable (e.g. the file is just one massive line), you can read one character at a time using read(size=1).

You can do something like this:

current = ""
with open("file.txt") as f:
    while True:
        char = f.read(1)
        if char == "":
            # Reached EOF
            break
        elif char.isdecimal():
            current += char
        else:
            num = float(current)
            # process num however you like
            current = ""

score 0 · Answer 4 · answered Jun 18 '21 at 13:09

You can try using the str.isspace() methods to check for spaces:

nums = ['0']
char = ' '
with open('file.txt', 'r') as f:
    while char:
        char = f.read(1)
        if nums[-1][-1].isspace():
            nums.append(char)
        else:
            nums[-1] += char
nums = list(map(float, nums))

Is there a way in Python to read text file token-by-token?

4 Answers4