Read from file while it is being written to in Python?

Question

I followed the solution proposed here

In order to test it, I used two programs, writer.py and reader.py respectively.

# writer.py
import time

with open('pipe.txt', 'w', encoding = 'utf-8') as f:
    i = 0
    while True:
        f.write('{}'.format(i))
        print('I wrote {}'.format(i))
        time.sleep(3)
        i += 1

# reader.py
import time, os

#Set the filename and open the file
filename = 'pipe.txt'
file = open(filename, 'r', encoding = 'utf-8')

#Find the size of the file and move to the end
st_results = os.stat(filename)
st_size = st_results[6]
file.seek(st_size)

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        print(line)

But when I run:

 > python writer.py

 > python reader.py

the reader will print the lines after the writer has exited (when I kill the process)

Is there any other way around to read the contents the time they are being written ?

[EDIT]
The program that actually writes to the file is an .exe application and I don't have access to the source code.

score 0 · Answer 1 · answered Mar 12 '20 at 14:13

0

You need to flush your writes/prints to files, or they'll default to being block-buffered (so you'd have to write several kilobytes before the user mode buffer would actually be sent to the OS for writing).

Simplest solution is to call .flush for after write calls:

    f.write('{}'.format(i))
    f.flush()

answered Mar 12 '20 at 14:13

ShadowRanger

143,180
12
188
271

This is not possible because I don't have the source code of the program that writes to the file. It is just an unknown ```.exe``` program. – Mar 12 '20 at 14:20
@entropyfeverone: Sigh. Then why did you describe it as a Python program? You might be able to make it work by wrapping execution with [the `stdbuf` command line utility](https://linux.die.net/man/1/stdbuf) (depends on how the `.exe` is implemented), but if that doesn't work, you're stuck. – ShadowRanger Mar 12 '20 at 14:41

score 0 · Answer 2 · answered Mar 12 '20 at 14:17

There are 2 different problems here:

OS and file system must allow concurrent accesses to a file. If you get no error it is the case, but on some systems it could be disallowed
The writer must flush its output to have it reach the disk so that the reader can find it. If you do not, the output stays in in memory buffer until those buffers are full which can require several kbytes

So writer should become:

# writer.py
import time

with open('pipe.txt', 'w', encoding = 'utf-8') as f:
    i = 0
    while True:
        f.write('{}'.format(i))
        f.flush()
        print('I wrote {}'.format(i))
        time.sleep(3)
        i += 1

This is not possible because I don't have the source code of the program that writes to the file. It is just an unknown ```.exe``` program — , Mar 12 '20 at 14:21

Read from file while it is being written to in Python?

2 Answers2

Linked