Reading file continuously and appending new lines to list (python)

Question

I am practicing file reading stuff in python. I got a text file and I want to run a program which continuously reads this text file and appends the newly written lines to a list. To exit the program and print out the resulting list the user should press "enter".

The code I have written so far looks like this:

import sys, select, os

data = []
i = 0

while True:

    os.system('cls' if os.name == 'nt' else 'clear')
    with open('test.txt', 'r') as f:
        for line in f:
            data.append(int(line))
            
    print(data)
    if sys.stdin in select.select([sys.stdin], [], [], 0)[0]:
        line_ = input()
        break

So to break out of the while loop 'enter' should be pressed. To be fair I just copy pasted the solution to do this from here: Exiting while loop by pressing enter without blocking. How can I improve this method?

But this code just appends all lines to my list again and again. So, say my text files contains the lines:

1
2
3

So my list is going to look like data = [1,2,3,1,2,3,1,2,3...] and have a certain lenght until I press enter. When I add a line (e.g. 4) it will go data = [1,2,3,1,2,3,1,2,3,1,2,3,4,1,2,3,4...].

So I am looking for some kind of if statement before my append command so that only the newly written line get appended. But I can't think of something easy.

I already got some tips, i.e.

1. Continuously checking the file size and only reading the part between old and new size.
2. Keeping track of line number and skipping to line that is not appended in next iteration.

At the moment, I can't think of a way on how to do this. I tried fiddling around with enumerate(f) and itertools.islice but can't make it work. Would appreciate some help, as I have not yet adopted the way of how programmers think.

Answer 1

Store the file position between iterations. This allows to efficiently fast-forward the file when it is opened again:

data = []
file_position = 0

while True:
    with open('test.txt', 'r') as f:
        f.seek(file_position)  # fast forward beyond content read previously
        for line in f:
            data.append(int(line))
        file_position = f.tell()  # store position at which to resume

Answer 2

I could get it to work on Windows. First of all, exiting the while-loop and continuous reading the file are two different questions. I assume, that exiting the while loop is not the main problem, and because your select.select() statement doesn't work this way on Windows, I created an exit-while with a try-except clause that triggers on Ctrl-c. (It's just one way of doing it).

The second part of your questions is how to continuously read the file. Well, not by reopening it again and again within the while loop, open it before the while loop.

Then, as soon as the file is being changed, either a valid or an invalid line is read. I suppose this happens because the iteration over f may sometimes happen before the file was completely written (I'm not quite sure about that). Anyway, it is easy to check the read line. I used again a try-except clause for it which catches the error if int(line) raises an error.

Code:

import sys, select, os

data = []

with open('text.txt', 'r') as f:
    try:
        while True:

            os.system('cls' if os.name == 'nt' else 'clear')
            for line in f:
                try:
                    data.append(int(line))
                except:
                    pass
                
            print(data)
    except KeyboardInterrupt:
        print('Quit loop')
print(data)