console.cs IsHandleRedirected oddity

Question

MS' own console.cs has the following function, used by the IsInput/Output/ErrorRedirected APIs.

private static bool IsHandleRedirected(IntPtr ioHandle) {

    // Need this to use GetFileType:
    SafeFileHandle safeIOHandle = new SafeFileHandle(ioHandle, false);

    // If handle is not to a character device, we must be redirected:
    int fileType = Win32Native.GetFileType(safeIOHandle);
    if ((fileType & Win32Native.FILE_TYPE_CHAR) != Win32Native.FILE_TYPE_CHAR) // <--- ??
        return true;

    // We are on a char device.
    // If GetConsoleMode succeeds, we are NOT redirected.
    int mode;
    bool success = Win32Native.GetConsoleMode(ioHandle, out mode);
    return !success;
}

I don't understand the logic on the line marked (by me) with // <--- ??. It would have made sense that if (fileType != Win32Native.FILE_TYPE_CHAR) return true;, but I don't follow why it's masked with & Win32Native.FILE_TYPE_CHAR before comparing.

To make it more confusing, the constant FILE_TYPE_CHAR is the single bit 0x0002 which is also shared by FILE_TYPE_PIPE = 0x0003, so the if statement in question will not return true; if the file handle refers to a pipe (maybe relying on GetConsoleMode to fail afterwards??).

Any insight into why that code was written the way it is would be much appreciated. Thanks.

Aww, shucks, they messed it up. Boo. Works by accident when ProcessStartInfo.CreateNoWindow = true, not uncommon for a redirected console mode app. Right way: https://stackoverflow.com/a/3453272/17034 — Hans Passant, Apr 22 '18 at 09:22
@HansPassant Thanks for verifying, and for the pointer. That's what I am doing in the end. I needed this in an applet targeted at .NET v4, which does not have those APIs, so I thought it would be "clever" to borrow them from the latest platform. But then I looked at the code and noticed the above. — dxiv, Apr 22 '18 at 16:20

Zohar Peled · Answer 1 · 2018-04-22T05:49:59.230

0

It used to be a common technique when working with flags enums.

You use bitwise and between the input value and the flag you want to check, and compare the results with the flag itself. If the input value contains the flag, the condition is true, if not, the condition is false.

Let me illustrate that with a simple example:

Suppose you have this enum:

[Flags]
enum test 
{
    None = 0, // 0000
    One = 1,  // 0001
    Two = 2,  // 0010
    Four = 4, // 0100
    Eight = 8 // 1000
}

And you have a value you want to test. Let's say 13. When you convert 13 to a four bit binary number you get 1101 - so you can do something like this:

var input = 13;
if(((test)input & test.One) == test.One) // that's testing if 1101 & 0001 = 0001
{
    // The result of this condition is true, since 13 is an odd number.
}

However, since .Net 4.0 you can simply use the HasFlag method instead:

if(input.HasFlag(test.One))
{
    // The result of this condition is true, since 13 is an odd number.
}

For more information you can read Ending the Great Debate on Enum Flags

edited Apr 22 '18 at 05:49

answered Apr 22 '18 at 05:33

Zohar Peled

79,642
10
69
121

Thank you, but I know what a flags enum is, and this one here is clearly *not* a flags enum. That was precisely the point of the 2nd paragraph in my question. The `FILE_TYPE_*` values are obviously *not* flags, since they share bits. The complete enum consists of values `0x0000, 0x0001, 0x0002, 0x0003, 0x8000` per the [official docs](https://msdn.microsoft.com/en-us/library/windows/desktop/aa364960(v=vs.85).aspx) (where the last one is documented as "*unused*"). – dxiv Apr 22 '18 at 06:01
That doesn't necessarily mean it's not a flags enum. It's perfectly valid for a flags enum to contain members that are a combination of other enum members. It might just means that `FILE_TYPE_PIPE` is a combination of `FILE_TYPE_DISK` and `FILE_TYPE_CHAR`, though I tend to agree it doesn't seem very likely. It might be just a code shortcut in this case. – Zohar Peled Apr 22 '18 at 06:13
`might be just a code shortcut` Or it could just be a code blunder ;-) I don't know of a "*file type*" that could qualify as both block (disk) and char, and I would certainly not call that a pipe if it existed. – dxiv Apr 22 '18 at 06:18

console.cs IsHandleRedirected oddity

1 Answers1