Enabling ANSI Color support in Windows console with custom screen buffer (C++)

Question

I'm trying to enable ANSI color support for created console screen buffer via CreateConsoleScreenBuffer().

hConsole = CreateConsoleScreenBuffer(GENERIC_READ | GENERIC_WRITE, 0, NULL, CONSOLE_TEXTMODE_BUFFER, NULL);

DWORD dwMode = 0;
GetConsoleMode(hConsole, &dwMode);
dwMode |= ENABLE_EXTENDED_FLAGS;
SetConsoleMode(hConsole, dwMode);

dwMode = 0;
GetConsoleMode(hConsole, &dwMode);
dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
SetConsoleMode(hConsole, dwMode);

SetConsoleActiveScreenBuffer(hConsole);

It seems like SetConsoleMode() isn't doing anything, I can write it to buffer as I intended, but if I try to write any ANSI Escape codes, it looks like this

ansii

If I'm not in buffer created by CreateConsoleScreenBuffer(), ANSI Escape codes are working as expected.

EDIT: I'm on Windows 10, 19041.388; C++14, MinGW-64 compiler

I vaguely recall `SetConsoleMode` and `ENABLE_VIRTUAL_TERMINAL_PROCESSING`. Assuming you are on current Windows 10. — Eljay, Aug 15 '20 at 13:10
Did you try what @Eljay suggested? `ENABLE_VIRTUAL_TERMINAL_PROCESSING` "_When writing with `WriteFile` or `WriteConsole`, characters are parsed for VT100 and similar control character sequences that control cursor movement, color/font mode, and other operations that can also be performed via the existing Console APIs. For more information, see Console Virtual Terminal Sequences._" — Ted Lyngmo, Aug 15 '20 at 13:28
A small question, you are trying to write with different colors in the console app is that true? — aliberro, Aug 15 '20 at 13:39
Tried putting it before `WriteConsoleOutputCharacterW()`, still problem persists, nothing changed. Seems like, it isn't going through VT100 parser? @TedLyngmo — Mayuna, Aug 15 '20 at 13:42

score 3 · Answer 1 · answered Aug 15 '20 at 14:01

You could use this:

#include <windows.h>
#include <iostream>
void Color(int color=0x07)
{
    SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), color);
}
int main()
{
    Color(0x0A/*Black bg green Fg*/);
    std::cout << "Hello";
    Color();
    std::cout << ",";
    Color(0xAC/*Green bg red Fg*/);
     std::cout << "World";
    Color(/*Black bg white Fg*/);
}

For more information about the colors:

Color attributes are specified by TWO hex digits -- the first corresponds to the background; the second the foreground. Each digit can be any of the following values:

0 = Black       8 = Gray
1 = Blue        9 = Light Blue
2 = Green       A = Light Green
3 = Aqua        B = Light Aqua
4 = Red         C = Light Red
5 = Purple      D = Light Purple
6 = Yellow      E = Light Yellow
7 = White       F = Bright White

Interestingly, it isn't changing text color while writing with `WriteConsoleOutputCharacterW()`. (used `SetConsoleTextAttribute(hConsole /* My buffer */, 0x0C);`) — Mayuna, Aug 15 '20 at 14:33
@Mayuna yes this would change the color before writing. Right after you call this function, everything that is written to the console would be in that color, it's a nice hack — aliberro, Aug 15 '20 at 14:36

score 0 · Answer 2 · answered Sep 25 '20 at 06:52

Virtual terminal sequences are control character sequences that can control cursor movement, color/font mode, and other operations when written to the output stream. Sequences may also be received on the input stream in response to an output stream query information sequence or as an encoding of user input when the appropriate mode is set.

You can use GetConsoleMode and SetConsoleMode functions to configure this behavior.

From ENABLE_VIRTUAL_TERMINAL_PROCESSING, we can use WriteFile or WriteConsole to achieve.

Some code:

HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);

HANDLE hConsole_c = CreateConsoleScreenBuffer(GENERIC_READ | GENERIC_WRITE, 0, NULL, CONSOLE_TEXTMODE_BUFFER, NULL);
SetConsoleActiveScreenBuffer(hConsole_c);
DWORD dwMode = 0;
GetConsoleMode(hConsole_c, &dwMode);
dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
SetConsoleMode(hConsole_c, dwMode);

const char* str = "\x1b[31mThis text has restored the foreground color only.\r\n";
DWORD len = strlen(str);
DWORD dwBytesWritten = 0;

WriteConsole(hConsole_c, str, len, &dwBytesWritten, NULL);

Debug:

Enabling ANSI Color support in Windows console with custom screen buffer (C++)

2 Answers2

Linked