Binary Pipe in Windows

Question

I'm able to create a child process and pipe its stdin and stdout. It works fine when it is in text mode. However, when I try to set the I/O in the child process to be in the binary format (i.e. no 0x0A to 0x0D 0x0A translation) the child process fails. _setmode returns -1 which has been documented to indicate failure. Why is that and how can it be fixed?

Parent code resembles the following:

const std::string path;  // = "path_to.exe"
PROCESS_INFORMATION info;
SECURITY_ATTRIBUTES sec_attr;

//in-out from the CHILD process' perspective
HANDLE out_r = nullptr;
HANDLE out_w = nullptr;
HANDLE in_r = nullptr;
HANDLE in_w = nullptr;


sec_attr.nLength = sizeof(SECURITY_ATTRIBUTES);
sec_attr.lpSecurityDescriptor = NULL;
sec_attr.bInheritHandle = TRUE; //inherit by child processes

if (!CreatePipe(&out_r, &out_w, &sec_attr, 0))
    throw std::exception();
if (!SetHandleInformation(out_r, HANDLE_FLAG_INHERIT, 0))
    throw std::exception();

if (!CreatePipe(&in_r, &in_w, &sec_attr, 0))
    throw std::exception();
if (!SetHandleInformation(in_r, HANDLE_FLAG_INHERIT, 0))
    throw std::exception();

if (out_r && out_w && in_r && in_w)
{
  startup_info.hStdError = out_w;
  startup_info.hStdOutput = out_w;
  startup_info.hStdInput = in_r;
  startup_info.dwFlags = STARTF_USESTDHANDLES;
}

if (CreateProcessA(path.c_str(), (char*)cmd, NULL, NULL, TRUE, 0, NULL, NULL, &startup_info, &info)
    != TRUE)
{
    DWORD error = GetLastError();
    //error handling
}

// ... read data using ReadFile

Child code resembles the following:

int status = _setmode(_fileno(stdin), _O_BINARY);
if (status == -1)
    throw std::exception();

status = _setmode(_fileno(stdout), _O_BINARY);
if (status == -1)
    throw std::exception();

puts("hello from the child process");

Which call to _setmode fails? The one for standard input, or the one for standard output, or both? What is the value of errno? — Harry Johnston, Apr 21 '17 at 22:20
Also, does the child process fail in the same way if you run it from the command line, e.g., `echo hello | child | findstr .` ? — Harry Johnston, Apr 21 '17 at 23:06
1) Both calls fail. 2) errno is 9, "bad file" 3) Runs fine when run from command line without piped in/out. Also runs fine as a native host for a Chrome extension, and when run from the command line. — Arya Pourtabatabaie, Apr 24 '17 at 14:12
I think at this point I need to provide more context. The parent process is a Native Unit Test. — Arya Pourtabatabaie, Apr 24 '17 at 14:14
You say it runs from the command line *without* piped input and output; does it run from the command line *with* piped input and output? (What I'm trying to get at is to determine whether the problem is in the child, or something about the way you're creating the pipes. If you let the command line create the pipes, and there's still a problem, the problem is in the child, otherwise it is in the parent.) But fix the bug Phil has pointed out first. — Harry Johnston, Apr 26 '17 at 10:38

score 2 · Accepted Answer · edited May 23 '17 at 11:46

I'm looking at the MSDN article Creating a Child Process with Redirected Input and Output that provides a reference implementation for nearly what you're trying to do here.

One subtle difference I see is, where in your parent code is written (the second call to SetHandleInformation()):

if (!SetHandleInformation(in_r, HANDLE_FLAG_INHERIT, 0))

the MSDN example (translated to use your variable names) would be written:

if (!SetHandleInformation(in_w, HANDLE_FLAG_INHERIT, 0))

Here are a couple more SO pages that would be worth looking at as well, if this small change doesn't help:

Binary Pipe in Windows

1 Answers1