5

The point was to make this little multiplayer game in the terminal, applying some basic graphics concepts to get a grasp on how it works and the maths behind it.

Note I wish to do this for fun and I am fully aware there are way better alternatives out there to using a terminal.

I'd need a console I could write to, so the point was to remove scroll bars and have the whole buffer printed to the screen.

But because of the carriage return when characters get written to the end of the previous line: carriage return

this would look overall ugly:

enter image description here

After trying for about 3 hours to get this working through SetConsoleMode, I asked about it in a discord.

The answer I got was this:

void main( )
{
    auto h = GetStdHandle( STD_OUTPUT_HANDLE );

    DWORD mode = 0;
    GetConsoleMode( h, &mode );

    mode |= DISABLE_NEWLINE_AUTO_RETURN | ENABLE_VIRTUAL_TERMINAL_PROCESSING;

    puts( SetConsoleMode( h, mode ) ? "Win" : "Loss" );

    char * buf = new char[ 200*2 ];
    memset( buf, 0, 200*2 );
    memset( buf, 'A', 120*2 );
    std::cout << ( buf );

    getchar( );
}

with this result: Code Result

At first I tried adapting it to my project, which failed. Then I got frustrated and ended up creating a new project into which I just copy pasted the working code.

And you guessed it, it doesn't work.

not working

Now I've tried quite a bit of stuff, I've changed my program from unicode to ansi and back, and done mode = DISABLE_NEWLINE_AUTO_RETURN | ENABLE_VIRTUAL_TERMINAL_PROCESSING, mode |= DISABLE_NEWLINE_AUTO_RETURN | ENABLE_VIRTUAL_TERMINAL_PROCESSING, and each individually with and without |with no success.

Given it's error 87 which means bad parameter, I even tried putting the exact value from GetConsoleMode to see if it was the handle, but since it worked we can assume it's not the handle's problem.

Basically code that works on another machine doesn't work on mine. The other machine was windows 10 mine's 8.1 both 64.

On both machines the value gotten off GetConsoleMode the first time is 3, which means ENABLE_PROCESSED_OUTPUT and ENABLE_WRAP_AT_EOL_OUTPUT.

The windows SDK version on my project is 10.0.15063.0, with a platform toolset of Visual Studio 2017 (v141).

I've been at this all day with no success. Now it wouldn't be the first time I read over some important detail, but I've been going through documentation and nothing mentions the failiure of SetConsoleMode using ENABLE_VIRTUAL_TERMINAL_PROCESSING.

As far as I can tell I'm doing it correctly based on the Docs docs

What should I be doing that I am not?

Sorry for any spelling/grammar mistakes that might have gone noticed and thank you for your time.

TrisT
  • 639
  • 6
  • 19
  • 1
    Not your fault, mind you. MSDN is usually pretty good at specifying minimum version requirements, but Microsoft really dropped the ball on this one. – Harry Johnston Sep 04 '17 at 04:54
  • 1
    Thank you both @eryksun and Harry Johnston (can't have 2 mentions in one comment). Would you mind making it into an answer so I can mark it solved? – TrisT Sep 04 '17 at 05:08

1 Answers1

8

Virtual terminal mode is available in the console starting with Windows 10.0.10586. It's not supported by the OS if setting the mode fails with ERROR_INVALID_PARAMETER (87). Also, it's only implemented in the new console. With the legacy console selected in Windows 10, enabling VT mode may succeed, but it won't actually enable VT support.

The failure case is mistakenly documented as "SetConsoleMode returning with STATUS_INVALID_PARAMETER" (0xC000000D) in the Enabling Virtual Terminal Processing example. It seems the writer overlooked that the kernel status code that's returned by the NtDeviceIoControlFile system call (in Windows 8+) gets translated to failing the SetConsoleMode call and translating the status code to a Windows API error code.

Eryk Sun
  • 33,190
  • 5
  • 92
  • 111
  • Is there a way to patch Win versions < W10 (e.g. W81) to be able to use VT? (Where else is this handled in the OS?) Perhaps by replacing the console with a statically compiled version? – not2qubit Dec 11 '18 at 12:01
  • @not2qubi, the console host is built into the OS. Prior to Windows 7 each console was hosted on a dedicated thread in the csrss.exe session server process. Since Windows 7 each console is hosted in a separated instance of conhost.exe. There are alternatives that hack into the console API to various extents, such as ANSICON (just hacks the API via DLL injection) and ConEmu (a powerful alternative console/terminal, which also relies on DLL injection). – Eryk Sun Dec 11 '18 at 12:17