1

I have a wrapper to redirect outputs when I call python python-wrapped C++.

The basic idea is to use dup and dup2, which are the only ways I've been able to catch the printf outputs from the C++. The wrapper works fine with no calls to flush() as long as I'm running the job interactively, but when I send the job to a TORQUE batch I get the unwelcome outputs again.

My understanding, in part from this question, is that some well-placed flush() calls should fix this, but where exactly do they need to go? Should I flush the buffer before dup'ing to the tempfile? Before dup'ing back? Both?

The wrapper I'm using is as follows: 

class Filter(object): 
    """
    Workaround filter for annoying and worthless errors. 
    """
    def __init__(self, veto_words={'ClassTable'}): 
        self.veto_words = set(veto_words)
        self.temp = tempfile.NamedTemporaryFile()
    def __enter__(self): 
        sys.stdout.flush() # <--- NEEDED? 
        sys.stderr.flush() # <--- NEEDED? 
        self.old_out, self.old_err = os.dup(1), os.dup(2)
        os.dup2(self.temp.fileno(), 1)
        os.dup2(self.temp.fileno(), 2)
    def __exit__(self, exe_type, exe_val, tb): 
        sys.stdout.flush() # <--- NEEDED? 
        sys.stderr.flush() # <--- NEEDED? 
        os.dup2(self.old_out, 1)
        os.dup2(self.old_err, 2)
        self.temp.seek(0)
        for line in self.temp: 
            veto = set(line.split()) & self.veto_words
            if not veto: 
                sys.stderr.write(line)
Community
  • 1
  • 1
Shep
  • 7,990
  • 8
  • 49
  • 71

1 Answers1

2

Python applies line buffering when connected to a TTY, otherwise a larger buffer is needed.

Redirecting your Python program to a pipe means there is no TTY connected to the stream, and you'll have to use .flush() even when sending newlines.

You can run Python with -u to turn off buffering of stdout.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • What I don't understand is how many times I have to call `flush()`. The example I linked to called it _before_ duping, whereas I'd expect that you'd call `flush()` before setting the stream _back_, since I want to avoid having the buffer dump into the original out/error stream. – Shep Mar 30 '13 at 16:34