0

I am writing a Windows application which does some console manipulation. First, I call GetStdHandle(STD_INPUT/OUTPUT_HANDLE), then GetConsoleMode/SetConsoleMode. However, if stdin/stdout are redirected, GetStdHandle will succeed, but then GetConsoleMode will return an 'invalid handle' error, presumably because it expects a console handle but I'm providing it with a file handle.

Currently I propagate such errors up, since I have no way of telling if such an error is 'real' (i.e., caused by some programming error or an internal error) or if they have been caused by this handle type mismatch.

Is there some way to tell what 'type' of handle I have? Alternatively, is there some way to tell that stdin/stdout have been redirected? If not, am I supposed to just ignore these particular errors?

Harry Wagstaff
  • 343
  • 2
  • 10
  • You actually can't. It's like a file handle in linux, completely opaque, unless you know how that handle was create by the operating system. – πάντα ῥεῖ May 13 '21 at 09:46
  • On Linux I can get an FD and then compare it against STDIN/STDOUT_FILENO, call isatty or other tty methods, or even use procfs to get a link back to the file or pipe that the fd points to. Is nothing like this possible on Windows? – Harry Wagstaff May 13 '21 at 09:51
  • 1
    you can use [`GetFileType`](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfiletype) – RbMm May 13 '21 at 10:01
  • [`GetStdHandle`](https://learn.microsoft.com/en-us/windows/console/getstdhandle#remarks) reference, @RbMm and this [thread](https://stackoverflow.com/questions/3453220/how-to-detect-if-console-in-stdin-has-been-redirected) all indicate that disambiguating the type between console, pipe, file, and others can be performed with `GetFileType` and [another thread](https://stackoverflow.com/questions/743885/how-can-i-determine-whether-console-out-has-been-redirected-to-a-file) covers more other details. – YangXiaoPo-MSFT May 14 '21 at 01:52
  • For `GetConsoleMode/SetConsoleMode` the right handles to use would be those returned by `CreateFile` for `CONIN$` and `CONOUT$`, not the `GetStdHandle` ones. See [console handles](https://learn.microsoft.com/en-us/windows/console/console-handles). – dxiv May 14 '21 at 03:54

0 Answers0