Four processes appending text to the same file

Question

I have 4 Python scripts / processes running in parallel, each one appending text to a file every ~30 seconds:

while True:
    id = processing_job() # 30 seconds long
    with open('log.txt', 'a+') as f:
        f.write('Job number #%i done.' % id)

Is there a risk, when using open(..., 'a+') that 2 processes want to write exactly at the same time, and then some text cannot be written in the log.txt and is lost?

Note: I'm using Windows platform.

I'm pretty sure that you will need some sort of synchronisation/serialisation. If these processes are created by `multiprocessing` then you could use a `Queue` to a single consumer which could perform the appending. — quamrana, Jan 23 '18 at 16:37
They are not been created with `multiprocessing` but just run in four different consoles (windows cmd.exe) — Basj, Jan 23 '18 at 16:42
The danger is not so much that data will be lost, as that the writes will be interleaved. — William Pursell, Jan 23 '18 at 17:08
@WilliamPursell what do you mean exactly? (Not sure to understand and why it would be so). The risk could be "Sorry file not writable, someone else is already writing in it', no? — Basj, Jan 23 '18 at 17:11
If a process attempts to write to a file while another process is writing, the write should block. That is, the 2nd process attempting the write will wait until the first finishes, and then proceed. If two process are trying to write data then they may alternate rapidly, resulting in the data being interleaved. — William Pursell, Jan 23 '18 at 17:36
@WilliamPursell Thanks. I think you can post an answer with this information. Also let's say process 1 has opened the file and is writing during 1 minutes (gigabytes of data...) is there a timeout after which process 2 will not wait anymore but say "Sorry file is busy, we can't write". — Basj, Jan 23 '18 at 18:05
If process 1 is writing gigabytes of data, the writes will not be atomic. (Please note, everything I say is from a unix perspective. I have no idea how/if windows actually works.) The operating system will allow a small (eg 4k) block of data to be written, and will then allow the other process to write a block. There may be some way to configure the system to timeout and discard data, but if such a (mis)feature were enabled, I would not consider the OS to be reliable. — William Pursell, Jan 23 '18 at 18:24
Possible duplicate of [Is file append atomic in UNIX?](https://stackoverflow.com/questions/1154446/is-file-append-atomic-in-unix) — georgexsh, Jan 31 '18 at 20:32
@Basj than you could take a look at https://stackoverflow.com/questions/3032482/is-appending-to-a-file-atomic-with-windows-ntfs — georgexsh, Feb 10 '18 at 17:52

F. Leone · Answer 1 · 2018-01-23T16:48:35.960

I did a quick test, opening the same file and it didn't give me any problem.

Python36 has been used.

Tested like so:

test1.py

with open("test","a+") as file_:
    file_.write("First process!\n")
print("done!")

test2.py

with open("test","a+") as file_:
    file_.write("Second process!\n")
print("done!")

python3 test1.py & python3 test2.py

The output is as expected.

First process! Second process!

Ran a lot of other variations (a bit string for the first and in the meanwhile the second) on the same file, didn't encounter any error

Four processes appending text to the same file

1 Answers1