Reading CSV file with python

Question

filename = 'NTS.csv'
mycsv = open(filename, 'r')
mycsv.seek(0, os.SEEK_END)

while 1:
   time.sleep(1)
   where = mycsv.tell()
   line = mycsv.readline()

if not line:
    mycsv.seek(where)
else:
    arr_line = line.split(',')
    var3 = arr_line[3]

    print (var3)

I have this Paython code which is reading the values from a csv file every time there is a new line printed in the csv from external program. My problem is that the csv file is periodically completely rewriten and then python stops reading the new lines. My guess is that python is stuck on some line number and the new update can put maybe 50 more or less lines. So for example python is now waiting a new line at line 70 and the new line has come at line 95. I think the solution is to let mycsv.seek(0, os.SEEK_END) been updated but not sure how to do that.

It depends on how the file is rewritten. If the file is effectively rewritten *in place*, I could not reproduce... except that as the pointer is at a given position in the file, it will only read again when the file will have at least that length. But beware: a pointer in a file has no notion of line count, only of byte count... — Serge Ballesta, Nov 16 '16 at 15:28

score 0 · Answer 1 · answered Nov 16 '16 at 15:28

0

You can do a simpler way of reading lines in your program. Instead of trying to use seek in order to get what you need, try using readlines on the file object mycsv.

You can do the following:

mycsv = open('NTS.csv', 'r')
csv_lines = mycsv.readlines()

for line in csv_lines:
    arr_line = line.split(',')
    var3 = arr_line[3]
    print(var3)

answered Nov 16 '16 at 15:28

irmiller22

172
6

Your solution is ok but I have around 1000 lines and every time I run the code python starts to read from the first line. I need python to start from the last. line – manev Nov 16 '16 at 15:41

score 0 · Accepted Answer · edited May 23 '17 at 12:34

What you want to do is difficult to accomplish without rewinding the file every time to make sure that you are truly on the last line. If you know approximately how many characters there are on each line, then there is a shortcut you could take using mycsv.seek(-end_buf, os.SEEK_END), as outlined in this answer. So your code could work somehow like this:

avg_len = 50  # use an appropriate number here
end_buf = 3 * avg_len / 2

filename = 'NTS.csv'
mycsv = open(filename, 'r')
mycsv.seek(-end_buf, os.SEEK_END)
last = mycsv.readlines()[-1]

while 1:

    time.sleep(1)
    mycsv.seek(-end_buf, os.SEEK_END)
    line = mycsv.readlines()[-1]

    if not line == last:

        arr_line = line.split(',')
        var3 = arr_line[3]

        print (var3)

Here, in each iteration of the while loop, you seek to a position close to the end of the file, just far back enough that you know for sure the last line will be contained in what remains. Then you read in all the remaining lines (this will probably include a partial amount of the second or third to last lines) and check if the last line of these is different to what you had before.

Reading CSV file with python

2 Answers2