Does Python's open() read an unflushed buffer?

Question

My knowledge of how file buffers work is fairly weak, so forgive the simple mindedness of this question. I have a piece of code in Python that waits for a file to appear, then reads it.

while 1:
    try:
        f = open('file.txt')
        f.close()
        print "Message received."
        break
    except:
        time.sleep(0.3)

Where file.txt is a file written by another program. However, I've a suspicion that Python is reading the file and then closing the handle before the file file.txt is completely written (that is, before I even call close() in the program that is writing the file). Should this even be possible?

If it is the case, is there a way I can detect (in the reading program listed above) whether or not the buffers have been flushed before I call f.close()?

In this case, no, because my goal in this little program is to check if the file exists, and is fully written. But would `f.read()` solve my problem? (the problem of unflushed buffers) — Gilead, Oct 13 '11 at 21:39

score 3 · Accepted Answer · edited May 23 '17 at 12:12

3

Don't use a while-loop for this purpose. There is a better way, though the details depend on your OS.

Linux has inotify. You can write a simple pyinotify script to watch the directory and react to IN_CLOSE_WRITE events (which occur only after the file has been closed).

OSX (FSEvents) and Windows have similar facilities.

edited May 23 '17 at 12:12

Community

1
1

answered Oct 13 '11 at 21:49

unutbu

842,883
184
1,785
1,677

Thanks, this was what I was looking for. I've used pyinotify to monitor for FS changes, but it didn't occur to me to use it for this very simple task. – Gilead Oct 13 '11 at 22:02

score 1 · Answer 2 · answered Oct 13 '11 at 21:40

Python will not stop you from reading a file that another process has not completed writing.

If I'm waiting for a file from another process, I usually have that process write to a different file name. Once that process has finished writing it closes and renames the file to the file name i'm expecting. That way I don't get a partial file.

score 1 · Answer 3 · answered Oct 13 '11 at 21:43

To check whether a file exists, use os.path.exists or maybe os.path.isfile if you require it to be a "regular" file.

On Unix, a directory entry will be created when the file is opened for write/append. But bytes written to a file will not actually show up there (from the point of view of a reader) until I/O is flushed. There is no normal way of asking whether there is "pending" I/O on a file; at least not on Unix.

score 0 · Answer 4 · answered Oct 13 '11 at 23:20

In general Linux has no problem whatsoever with one process opening a file for read while another process has it open for writing. Think about doing things like tail -f $SOME_FILE to follow updates to a file in real time; that could not possibly work if attempting to open the file for reading had to block and wait until the writer closed the file.

Does Python's open() read an unflushed buffer?

4 Answers4