0

Below is part of a script that I wrote which takes each item in a list, passes it to a function, stores the output of the function as a variable with a line break at the end and then writes the variable to a new line in a file. It also prints the variable to the console.

for i in lane_list:
        out = count_genes(count, i, reader, total_genes) + '\n'
        count += 1
        outfile.write(out)
        print out

The script works but not in the way that I would expect. I can see the output being printed to the console as the script runs so I know the rate at which the script is running through the for loop and I assumed that python would write to the file at the same rate as it prints output to the console. What's odd is that for long periods of time nothing will be written to the file (I can see this by 'cating' the file in the console as the script's running) and then several hundred more lines will appear in the file at once. It's as if python is storing what it has to write to the file for a period of time and then writing all of it in one go. What was even more surprising is sometimes when I cat the file as the scripts running it will have written half of the last line but not all of it.

Can anyone explain to me why this is?

Ben Jeffrey
  • 714
  • 9
  • 18
  • 4
    Read about flushing: http://stackoverflow.com/questions/3167494/how-often-does-python-flush-to-a-file. Also it maybe helpful to use `tail -f filename` to get live preview of file (if you are working on linux). – Szymon P. Mar 23 '17 at 14:10

1 Answers1

1

As @szymon mentioned, the file is buffered as it is written. If you want to see it being written 'live', use open('/path/to/file'', 'w', 0) where the 0 means unbuffered.

For python3 use 1 for line buffered as unbuffered is only allowed for binary data.

Holloway
  • 6,412
  • 1
  • 26
  • 33
  • Thanks all, is there any reason not to use this? Will it slow down for example? – Ben Jeffrey Mar 23 '17 at 14:17
  • It can slow down - every time you save to the file it hits the disk (I/O operations). When buffer size is set, script hits the disk only when buffer is full (and flushes it). – Szymon P. Mar 23 '17 at 14:22