Win32 changing to binary mode child's Stdout (pipe)

Question

Hello to this great community,

I have problems with the automatic conversion of ('\n') 0x0A to ('\n\r') 0x0D 0x0A when using a pipe to redirect child's stdout to a file, the child's output are bytes and not text.

First, I have used these examples MSDN-Creating a Child Process with Redirected Input and Output and http://support.microsoft.com/kb/190351), and now I have this basic application, it creates a pipe and redirect the child's STDOUT to a binary file. All this in a Win32 Console Application in Visual C++ 6.0 (yes it is old but is a requirement).

#define BUFSIZE 256

HANDLE g_hChildStd_OUT_Rd = NULL;
HANDLE g_hChildStd_OUT_Wr = NULL;

int _tmain(int argc, TCHAR *argv[]) 
{ 

    SECURITY_ATTRIBUTES saAttr; 
    saAttr.nLength = sizeof(SECURITY_ATTRIBUTES); 
    saAttr.bInheritHandle = TRUE; 
    saAttr.lpSecurityDescriptor = NULL; 

    if ( ! CreatePipe(&g_hChildStd_OUT_Rd, &g_hChildStd_OUT_Wr, &saAttr, 0) ) 
        ErrorExit(TEXT("StdoutRd CreatePipe")); 

    if ( ! SetHandleInformation(g_hChildStd_OUT_Rd, HANDLE_FLAG_INHERIT, 0) )
        ErrorExit(TEXT("Stdout SetHandleInformation")); 

    CreateChildProcess();

    if (!CloseHandle(g_hChildStd_OUT_Wr)) 
        ErrorExit("CloseHandle");

    ReadFromPipe(); 

    if (!CloseHandle(g_hChildStd_OUT_Rd)) 
        ErrorExit("CloseHandle");

    return 0; 
} 


void CreateChildProcess()
{ 
    TCHAR szCmdline[]=TEXT("child.exe");
    PROCESS_INFORMATION piProcInfo; 
    STARTUPINFO siStartInfo;
    BOOL bSuccess = FALSE; 

    ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION) );

    ZeroMemory( &siStartInfo, sizeof(STARTUPINFO) );
    siStartInfo.cb = sizeof(STARTUPINFO); 
    siStartInfo.hStdOutput = g_hChildStd_OUT_Wr;
    siStartInfo.dwFlags |= STARTF_USESTDHANDLES;

    bSuccess = CreateProcess(NULL, 
        szCmdline,  // command line 
        NULL,       // process security attributes 
        NULL,       // primary thread security attributes 
        TRUE,       // handles are inherited 
        0,      // creation flags 
        NULL,       // use parent's environment 
        NULL,       // use parent's current directory 
        &siStartInfo,   // STARTUPINFO pointer 
        &piProcInfo);   // receives PROCESS_INFORMATION 

    if ( ! bSuccess ) 
        ErrorExit(TEXT("CreateProcess"));
    else 
    {
        CloseHandle(piProcInfo.hProcess);
        CloseHandle(piProcInfo.hThread);
    }
}


void ReadFromPipe(void) 
{ 
    DWORD dwRead, dwWritten; 
    CHAR chBuf[BUFSIZE]; 
    BOOL bSuccess = FALSE;
    HANDLE hParentStdOut = GetStdHandle(STD_OUTPUT_HANDLE);

    DWORD nTotalBytesRead = 0;
    fstream filePk;
    filePk.open("result.out", ios::out | ios::trunc | ios::binary);

    for (;;) 
    { 

        bSuccess = ReadFile( g_hChildStd_OUT_Rd, chBuf, BUFSIZE, &dwRead, NULL);
        if( ! bSuccess || dwRead == 0 ) {
            if (GetLastError() == ERROR_BROKEN_PIPE)
                break; // pipe done - normal exit path.
            else
                ErrorExit("ReadFile"); // Something bad happened.
        }

        filePk.write(chBuf, dwRead);
        nTotalBytesRead += dwRead;
    } 
    filePk.close();

    char ibuff[24]; 
    sprintf(ibuff,"%d bytes." , (int)nTotalBytesRead);
    ::MessageBox(NULL, ibuff, "", 0);
} 

And in this dummy child.cpp you'll notice that if I set the STDOUT to binary mode everything works just fine (I get just 0x0A 0x0A!), but my real child is an EXE and I don't have access to that code.

int main(int argc, char* argv[])
{
    _setmode( _fileno( stdout ), _O_BINARY );
    printf("\n");

    unsigned char buffer[] = {'\n'};
    fwrite(buffer, sizeof(unsigned char), sizeof(buffer), stdout);
    return 0;
}

So, after searching for about 2 days and considering that I have a basic C++ knowledge I ask: Is there a Way that I could do _setmode to the childs stdout from the parent, considering that I don't have access to the child's code.

As a solution, I am seriously considering finding every '0x0D' '0x0A' and replacing it with '0x0A'. I am really going crazy with this problem... So if someone could help me I will be very grateful.

Related Question: Win32 Stream Handles - Changing To Binary Mode but he has access to the child's code!

Edit

As, @librik point, the final solution will have to replace every single occurrence of 0x0D 0x0A by 0x0A. For this to work the file contents must be in memory. There are certain issues but I can live with it (excess of memory allocated). I hope this will be helpful:

void ReadFromPipe(void) 
{ 
    DWORD dwRead, dwWritten; 
    CHAR *chBuf = NULL, *chBufTmp = NULL; 
    BOOL bSuccess = FALSE;
    HANDLE hParentStdOut = GetStdHandle(STD_OUTPUT_HANDLE);

    DWORD nTotalBytesRead = 0;
    fstream filePk;
    filePk.open("result.out", ios::out | ios::trunc | ios::binary);

    int nIter = 0;
    for (;;) 
    {
        if(chBuf == NULL) {
            if((chBuf = (CHAR*)malloc(BUFSIZE*sizeof(CHAR))) == NULL) {
                ErrorExit("Malloc");
            }
        } else {
            chBufTmp = chBuf;   // save pointer in case realloc fails
            if((chBuf = (CHAR*)realloc(chBuf, (nIter+1)*(BUFSIZE*sizeof(CHAR)))) == NULL) {
                free(chBufTmp); // free original block
                ErrorExit("Realloc");
            }
        }

        CHAR* chBufNew = chBuf+nTotalBytesRead;
        bSuccess = ReadFile(g_hChildStd_OUT_Rd, chBufNew, BUFSIZE, &dwRead, NULL);
        if( ! bSuccess || dwRead == 0 ) {
            if (GetLastError() == ERROR_BROKEN_PIPE) {
                break; // pipe done - normal exit path.
            } else {
                ErrorExit("ReadFile"); // Something bad happened.
            }
        }

        nTotalBytesRead += dwRead;
        nIter ++;
    } 

    // 0xD 0xA -> 0xA
    nTotalBytesRead = ClearBuffer(chBuf, nTotalBytesRead);

    filePk.write(chBuf, nTotalBytesRead);
    filePk.close();
    free(chBuf);

    char ibuff[24]; 
    sprintf(ibuff,"%d bytes." , (int)nTotalBytesRead);
    ::MessageBox(NULL, ibuff, "", 0);
} 

int ClearBuffer(char *buffer, int bufferlength) {
    // lmiguelhm-es requerido que TODO el buffer esté en memoria 
    int chdel = 0;
    for (int i = 0; (i+chdel) < bufferlength; i++) {
        char firstChar = buffer[i+chdel];
        buffer[i] = firstChar;
        if (firstChar == 0x0D) {
            if ((i+chdel+1) < bufferlength) {
                char secondChar = buffer[i+chdel+1];
                if (secondChar == 0x0A) {
                    buffer[i] = secondChar;
                    chdel++;
                }
            }
        }
    }
    return bufferlength - chdel;
}

Answer 1

Your problem is that the "stream mode" isn't part of Windows, so it isn't something you can change from outside the other program. It's part of the C and C++ system, and so it's a private part of each separate C or C++ program you run.

There's a library of functions that is combined with every program compiled in C++, called the "C++ Standard Library." The C++ Standard Library holds all the functions for streams like stdout. It's inside the other program's C++ Standard Library that the 0x0A is being converted to 0x0D 0x0A before it's written to the stream. _setmode is a function inside the C++ Standard Library which turns on and off that conversion, so when you add a call to it inside child.cpp, that tells child.cpp's C++ Standard Library to leave stdout alone. But you have no way to force the other program to call its _setmode function.

So the best thing really is the "crazy" solution you suggested:

As a solution, I am seriously considering finding every '0x0D' '0x0A' and replacing it with '0x0A'.

So long as you know that child.exe is writing in text mode, not binary mode, then every single occurrence of 0x0D 0x0A must have originally been a single 0x0A. (If the program tried to write the two bytes 0x0D 0x0A, it would come out as the three bytes 0x0D 0x0D 0x0A.) Therefore you are absolutely safe and correct to "fix" the output by converting back again.

I think the easiest approach is just to write result.out exactly like you're doing it now, but then translate 0x0D 0x0A to 0x0A at the end, creating a new file that is correct. There are little tool programs you can download that will do this sort of thing for you -- one of them is called dos2unix. That might be the easiest way, in fact -- just make the last step of your program run dos2unix < result.out.with_bad_newlines > result.out. If, for some reason, you can't do this, you could have your program change 0x0D 0x0A to 0x0A inside chBuf before you write it out, translating as you go. (But be careful when chBuf ends in 0x0D...)

(There are certain techniques that can "inject" a little bit of code into another program that's under your control on Windows. They are a little dangerous, and a lot of trouble. If you're really unhappy with the translation idea, you can look up "DLL injection.")