In Powershell, how do you redirect a binary file to standard in?

Question

This one does not work for binary files: Redirecting standard input\output in Windows Powershell

Here's a summary of what I'm using for input. You can see ls shows the file is 858320 bytes, but when piped through Get-Content, this is not the case.

PS C:\Users\Joseph\Documents\GitHub\java_hprof_to_txt> ls .\generate_hprof\heap.dump.hprof


    Directory: C:\Users\Joseph\Documents\GitHub\java_hprof_to_txt\generate_hprof


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        1/28/2017  12:02 PM         858320 heap.dump.hprof

Here is my minimal test program which counts the bytes from standard input until it gets to eof:

#include <stdio.h>  
#include <fcntl.h>  
#include <io.h> 


int main()
{
    char buff;
    int numOfBytesRead;
    int dataLen = 0;

    #ifdef _WIN32
    int result = _setmode(_fileno(stdin), _O_BINARY);
    if (result == -1)
        perror("Cannot set mode");
    else
        printf("'stdin' successfully changed to binary mode\n");
    #endif

    while (true) {
        numOfBytesRead = fread(&buff, 1, 1, stdin);
        if (numOfBytesRead != 1) {
            if (feof(stdin)) {
                fprintf(stdout, "end of file reached\n");
            }
            int errorCode = ferror(stdin);
            if (errorCode) {
                fprintf(stdout, "error reading file %d\n", errorCode);
            }
            perror("The following error occurred:");
            break;
        }
        dataLen++;
    }
    fprintf(stdout, "read %d bytes\n", dataLen);
    return 0;
}

Here's the output. Notice that the bytes to not match with ls.

PS C:\Users\Joseph\Documents\GitHub\java_hprof_to_txt> Get-Content .\generate_hprof\heap.dump.hprof | .\x64\Debug\CountBytes.exe
'stdin' successfully changed to binary mode
end of file reached
The following error occurred:: No error
read 860183 bytes

I've even tried the -Encoding Byte and -Encoding Unknown but that doesn't help:

PS C:\Users\Joseph\Documents\GitHub\java_hprof_to_txt> Get-Content -Encoding Byte  .\generate_hprof\heap.dump.hprof | .\x64\Debug\CountBytes.exe
'stdin' successfully changed to binary mode
end of file reached
The following error occurred:: No error
read 3253650 bytes

PS C:\Users\Joseph\Documents\GitHub\java_hprof_to_txt> Get-Content -Encoding Unknown  .\generate_hprof\heap.dump.hprof | .\x64\Debug\CountBytes.exe
'stdin' successfully changed to binary mode
end of file reached
The following error occurred:: No error
read 429608 bytes

When I run it in a normal command terminal, it works fine:

C:\Users\Joseph\Documents\GitHub\java_hprof_to_txt>.\x64\Debug\CountBytes.exe <  .\generate_hprof\heap.dump.hprof
'stdin' successfully changed to binary mode
end of file reached
The following error occurred:: No error
read 858320 bytes

From the doc: "When reading from and writing to binary files, use a value of Byte for the Encoding dynamic parameter and a value of 0 for the ReadCount parameter." Would that help? — David Brabant, Feb 02 '17 at 06:55
Yeah I saw that, but I didn't want to load the entire file contents into memory because in the future the binary file can be large. I'll try that for now. I'm guessing my usecase doesn't align with the powershell philosophy of operating on objects instead of operating on steams. — joseph, Feb 02 '17 at 17:08
Don't use high-level cmdlets for low-level stuff, instead use .NET binary readers/writers and access the std streams directly, see [Output binary data on powershell pipeline](//stackoverflow.com/a/24745250) — wOxxOm, Feb 04 '17 at 02:37
That post is about the output side. I'm confused about the input since powershell doesn't support "<" — joseph, Feb 07 '17 at 20:09
@joseph `Get-Content` reads lines and omits CR/LF bytes. This is suitable for text files as the result will be an array of strings that represent a line each. To pipe the original data (so the binary data in this case), add the `-Raw` switch to `Get-Content`. — stackprotector, Jun 23 '20 at 10:44

score 5 · Accepted Answer · answered Jun 23 '20 at 11:06

If the addition of the -Raw parameter does not help:

Get-Content .\generate_hprof\heap.dump.hprof -Raw | .\x64\Debug\CountBytes.exe

Using the Start-Process cmdlet will work for sure:

Start-Process .\x64\Debug\CountBytes.exe -RedirectStandardInput .\generate_hprof\heap.dump.hprof

In Powershell, how do you redirect a binary file to standard in?

1 Answers1