Appropriate way to read a constantly updating (20 times per second) file in Python 3?

Question

I'm working with a csv file that's constantly growing, with about 20 lines being added per second. Each line needs to be parsed. The code snippet I have below does work, but it seems to stop updating after a bit. It's running in its own thread and if I manually update the csv file (ie. a new line every few seconds), it seems to work perfectly fine.

file=open('data.csv', 'r')
while True:
    line=file.readline()
    if len(line) > 2:
        print(line)
        #parse

This is on Ubuntu 14.04 and Python 3.5 (unfortunately, I'm stuck with these versions). Strangely enough, I haven't noticed any issues when running on Windows 7. Is there a better way to approach this?

Does the other side update the file in a single syscall, or does a single consists of multiple syscall (in what case you manually need to merge the line back together) — Ferrybig, Jan 16 '19 at 07:45
@RockBlazer Then it's possibly duplicate with https://stackoverflow.com/questions/12523044/how-can-i-tail-a-log-file-in-python — duong_dajgja, Jan 16 '19 at 07:48
@Ferrybig That's a good question, I'll need to check because I didn't write the other program. Should know in a few hours, but I think you might be on to something. — Rock Blazer, Jan 16 '19 at 07:51
@duong_dajgja Fair point. I'll try out tail and report back if I run into the same issue. Thanks. — Rock Blazer, Jan 16 '19 at 07:57

score 1 · Answer 1 · answered Jan 16 '19 at 07:52

Since OP has claimed that the file is appended then I would suggest to try answers to How can I tail a log file in Python?. In short, you could do:

Your python script should use the built-in tail and check return from the tail.
Use an external python module named sh

Appropriate way to read a constantly updating (20 times per second) file in Python 3?

1 Answers1