Writing to a file from a subprocess and parent process simultaneously

Question

I have a script (Python) that uses subprocess to call another script. The parent script writes to the console and a log file at the same time (I used the code from the accepted answer to this question to split the output), but the child process print statements are only going to stdout, not to the log file.

I do know why this is: the subprocess has stdout set to normal stdout, not the special tee object that the parent process has. I tried passing the tee object as an argument to the subprocess, but I learned the hard way that you cannot pass objects as arguments to subprocesses. My backup plan was to pass in the path string for the file to write to, and then have the subprocess make its own tee object with the same file.

My question is, if both processes are writing to the same file at the same time, will the output be messed up? I'm using open("file", 'w') in the parent process, and this is called first, and I'm using open("file", 'a') in the child process. Hypothetically, the file should contain the output of the print statements in the correct order, because appending to a file with 'a' means that the lines will always be added to what is currently the end of the file, will it not? Or are there rules about opening a file that prevent it from being opened by 2 processes at the same time?

POST_TEST: after doing some tests myself, I found the following: -You are allowed to open("file", 'w') multiple times in a row -You are allowed to open("file", 'w') and then open("file", 'a') -In the first case, the child process overwrites the file completely. -In the second case, the order is not correct, and some output seems to be lost.

My new question then, is what alternative solution should I use to write to a file from both the parent and child process at the same time, without getting lines out of order or overlapping?

Google "mutex". A mutex is a MUTually EXclusive lock on a resource, that avoids collisions when two processes use the same resource. — Juan Tomas, Jun 30 '16 at 21:20
http://stackoverflow.com/questions/489861/locking-a-file-in-python — a_river_in_canada, Jun 30 '16 at 21:20
The process that opens in `a` mode will always write at the end of the file. But the one that opens in `w` mode will not. You need to open them both in `a` mode. — Barmar, Jun 30 '16 at 21:22
@JuanTomas I actually learned about mutexes last year in a C++ class, didn't think of using one here. Thanks for the tip — Eliezer Miron, Jun 30 '16 at 21:22
@Barmar That actually worked perfectly, but what if the file already exists? — Eliezer Miron, Jun 30 '16 at 21:25
Call `file.truncate()` after you open it in the parent process. — Barmar, Jun 30 '16 at 21:28

score 0 · Accepted Answer · edited May 23 '17 at 12:32

0

Thank you Bamar for the suggestions: using 'a' for both the first and second open() works. If you need to do this for a file that already exists, you can use file.truncate() to empty the file before appending to it.

edited May 23 '17 at 12:32

Community

1
1

answered Jun 30 '16 at 21:32

Eliezer Miron

388
5
18

It is not guaranteed that all writes are at the end on all systems if the file is opened in `'a'` mode. It is not guaranteed that larges `write()`s are atomic (the output from several processes may interleave). – jfs Jul 01 '16 at 14:15

score 0 · Answer 2 · edited May 23 '17 at 12:14

Opening a file in 'w' mode truncates the file (as documented). Opening a file in 'a' mode may work on some systems. POSIX says for O_APPEND flag:

If set, the file offset shall be set to the end of the file prior to each write.

write()s that are larger than PIPE_BUF may interleave:

POSIX.1-2008 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic.

Q: what alternative solution should I use to write to a file from both the parent and child process at the same time, without getting lines out of order or overlapping?

Open the files using line-buffering mode (1), to flush the internal buffer at the end of each line -- it should preserve the approximate relative order of the lines. If lines are less than PIPE_BUF (4096 bytes on my system) then they should not "overlap".

The subprocess may write data to its standard output and your parent Python process may write it to a file in any order you like. See how teed_call() function is implemented.

I don't mind the downvote but I would appreciate an explanation that would allow to make the answer better. — jfs, Jul 03 '16 at 16:16
I hate it when people downvote without saying why they're doing it — Eliezer Miron, Aug 16 '16 at 19:24

Writing to a file from a subprocess and parent process simultaneously

2 Answers2