How to manage subprocess in Python when the output is huge?

Question

I'm controlling long running simulations (hours, days, even weeks) using a bash script that iterates over all wanted parameters. If only one simulation runs concurrently, the output is piped to "tee", else the output is plainly piped ">" to an output file. All output are huge: some log files are ~2GB and could be even bigger.

The script is working, but is a hell to maintain. When we add a new parameter it takes some time to adapt the script and all the sed-foo in it. So I've ported it to Python. It's working GREAT.

The only problem I have now preventing me from using it in production is that I can't find the right way of calling Popen() to launch the program. If I run it "silent" by piping everything to the file and not showing any output, python takes gigabytes of ram before the simulation is done.

Here's the code snipet:

fh = open(logfile, "w")
pid = subprocess.Popen(shlex.split(command), stdout=fh)
pids.append(pid)

I've read a lot of stuff about Popen the output, but I though that piping it to a file would flush the buffer when needed?

Maybe subprocess' Popen() is not the best for this? What's the best way to show and save a program's output to screen and file without taking all the ram?

Thanx!

Answer 1

Why not write silently to a file and then tail it?

You can use file.flush() to clear Python's file buffer.

---

Python will happily handle new lines in a currently-open file. For instance:

f = open( "spam.txt", "r" )
f.read()
# 'I like ham!'
# Now open up spam.txt in some other program and add a new line.
f.read()
# 'I like eggs too!'

Answer 2

Instead of splitting the output of your simulation, choose to pipe it into a file (or write to a file from within the simulation, then use tail -f to watch the latest output in a console.

Example simulation:

#!/bin/bash

while true; do
  echo $$ -- $(date +%s)
  sleep 1
done

Or perhaps:

#!/usr/bin/env python
import os, sys, time

while True:
  sys.stdout.write("%d -- %d\n"%(os.getpid(), time.time()) )
  sys.stdout.flush()
  time.sleep(1)

Invocation:

$ nohup ./simulation &> logfile &

Watching the output:

$ tail -f logfile
1285 -- 1337166243
1285 -- 1337166244
1285 -- 1337166245
1285 -- 1337166246
1285 -- 1337166247
^C

Notes:

• Bonus points for splitting stderr and stdout to different logfiles.
• Don't use tee for things like this. It's fragile and will propagate errors to your simulation in case something bad happens at the pipe's end.
• Note how we record the PID of the simulation so that we can abort it if we want after it has been started. it's recommended that you store this in a pidfile instead of the simulation log, purely for simplicity when killing your simulation.
• Use nohup. This will protect your simulation run in case you close the originating terminal, or if X11 crashes (from experience, this will happen when your 4 day simulation is 98% complete, and you haven't implemented checkpoints...).

Answer 3

The simplest solution was to change the code so it outputs to stdout AND a log file. Then, the output does not neet to be saved using tee or a pipe.

pipe_verbose = sys.stdout
pipe_silent  = open('/dev/null', 'w')

subprocess.Popen(shlex.split(command), stdout=pipe_silent)
subprocess.Popen(shlex.split(command), stdout=pipe_verbose)

and finally I poll() to see when done.

Piping has the nice result of that if I ctrl+c the script, it kills the job too. If I did not put stdout=... in the Popen(), then the job continues in the background. Also, python's CPU usage stays at 0% that way. A readline loop on a pipe would raise it to 100%...

Answer 4

If output has reliably-occurring output delimiters (markers indicating end of output section) consider doing the "bad" thing and reading the stdout chunks from subprocess in a separate thread and writing individual chunks to log, flashing them with every write.

Take a look here for some examples of non-blocking reads from subprocess' pipe:

How can I read all availably data from subprocess.Popen.stdout (non blocking)?