Repeated redirection of low-level stdin in Python

Question

Here is my situation: There is some old C++ code which I am attempting to create a simple Python wrapper for. This code was originally designed to be driven from the command line, and is expecting an input file to be piped into it. In the interests of minimally interfering with this code, I have converted the main function into a vanilla extern "C" function and compiled it into a shared library, which I load up in Python and drive via ctypes. Note that I don't want to drive this code via system calls since it is going to end up running on a cluster and executing zillions of times.

From here, I need to take control of the lower level stdin in order to set up the input file so that it can be read via various "cin" statements in the C++ code.

From what I have learned so far, it seems I can in theory do this by overwriting the stdin file descriptors using the os library. I cooked up the following example which seems to work perfectly well in Python (based on various information in this thread: Python version of freopen()):

import os
import sys

def freopen(f,option,stream):
    oldf = open(f,option)
    oldfd = oldf.fileno()
    newfd = stream.fileno()
    os.close(newfd)
    os.dup2(oldfd, newfd)

# Original stdout file descriptor:
fd = sys.stdout.fileno()
orig_stream = os.fdopen(os.dup(fd), 'w') 

# Test writing to file:
freopen("hello","w",sys.stdout)
print "world"
sys.stdout.flush()
freopen("hello2","w",sys.stdout)
print "world2"
sys.stdout.flush()

# Restore stdout to normal
os.dup2(orig_stream.fileno(),sys.stdout.fileno())

print "back to normal!"

# Test reading:
freopen("hello","r",sys.stdin)
print sys.stdin.readlines()

freopen("hello2","r",sys.stdin)
print sys.stdin.readlines()

This produces the output

back to normal!
['world\n']
['world2\n']

(as well as the two files "hello" and "hello2") for me. So great! Unfortunately it doesn't seem to work when the stdin stream is being read by my C++ library function. Weirdly, it works the first time just fine, but the second time it fails; no error occurs, it seems more like the stdin stream is simply empty for the second redirection. When I redirect the stdout instead of stdin it seems to work fine.

Is there something special I need to do to reset the lower level stdin stream for the second pass?

Extra info: On the c++ side the stdin is just being read like so:

while (getline(cin,line)) {
        //    do stuff with line...

Also I am not too concerned if the solution is "Linux only".

Edit: I am running Python 2.7.1

Update: Hmm, maybe the library code is doing something odd, because if I change the "reading test" part of my test code to this:

# Test reading:
freopen("hello","r",sys.stdin)
os.system("cat")

freopen("hello2","r",sys.stdin)
os.system("cat")

Then the expected output

back to normal!
world
world2

is still produced. So however 'cat' is treating the stdin seems perfectly compatible with this method of stdin redirection. Hmm. Of course a new "cat" process is created via this method and new stdin hooked up to it each time, so it is not really the same as calling my library function. I will try to create a minimal library function which reproduces the problem...

When you say it works "the first time" what do you mean exactly? The first line only, the first time before restarting Python, or...? — John Zwinck, Nov 15 '13 at 11:47
Ahh sorry. I mean the first time I perform the redirection + run the library function. The whole library function performs as expected. However, if a couple of python lines later I repeat the process, redirection stdin to a new file as in my example, the library function seems to receive no input from its cin calls and fails. — Ben Farmer, Nov 15 '13 at 11:50
Basically replace the "print sys.stdin.readlines()" in my python example with a call to the library function to get what I mean. — Ben Farmer, Nov 15 '13 at 11:52

score 0 · Accepted Answer · edited May 23 '17 at 12:05

Aha! So as it turns out, this question was of great help: cin.ignore() and cin.clear() in C++

So it seems that in C++ at least, when a stream reaches an eof then it sets various error flags to true (http://www.cplusplus.com/reference/ios/ios/good/). This prevents further operations from being performed on the stream. When my library code was a stand-alone executable, this behaviour was fine because the program would only try to read a file from cin once. However, with code converted into a library, when my loop returned to calling the library function a second time, the library function was still dealing with the same stdin stream. The stream still had the error flags set, so the cin reads were failing on this second pass.

To solve the problem, I added

std::cin.clear();

to the library code, before the cin reads were attempted. This resets the error flags for the stream, which could then be read just fine.

I am not sure if this resetting of stream flags can be dealt with on the Python side or not; this would be good to know.

Repeated redirection of low-level stdin in Python

1 Answers1