Calling to system function in C causes it to fix ANSI colors?

Question

I've been making tools for the command line for a little while now. However, one thing that stuck out to me when I was going over some old projects was this:

// I ABSOLUTELY NEED THIS BUT IDK WHY!!!!
system("Title Something");

So I decided to investigate. When I removed it and ran the tool again in a simulated environment (more specifically VSCodes simulated console) everything seemed fine. However when I tried running it in cmd I saw that all of the colors that I had previously used in my code where broken.

So I pulled up a temporary projects like this one:

#include <stdio.h>
int main(){
  printf("\033[1;31mI am red!\033[0m");
}

And when I ran it I got:

C:\tests\ANSI-test>hello
←[1;31mI am red!←[0m

Which was similar to what I had previously gotten with my other tool. And so I added back the seemingly random system("Title Something"); line (making sure I included stdlib.h):

#include <stdio.h>
#include <stdlib.h>
int main(){
  system("Title Something");
  printf("\033[1;31mI am red!\033[0m");
}

And I got color working again!:

C:\tests\ANSI-test>hello
I am red!                                                   (pretend this is red)

I'm not sure exactly what causes this, I have a couple of speculations:

• the system function initializes something to do with the console, which triggers the working of ANSI colors
• system somehow calls to WinAPI and tells it to use colors inside the terminal

I did try searching for this but I didn't really find anything related to it other than this:

https://stackoverflow.com/questions/16755142/how-to-make-win32-console-recognize-ansi-vt100-escape-sequences-in-c#:~:text=Starting%20from%20Windows%2010%20TH2,they%20have%20to%20be%20enabled).

Answer 1

Interesting find. From what I am able to gather, system() seems to set the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag (I am not sure if it does this before or after forking, but thinking about it more, I don't think this would matter since its setting it for the entire console handle). I find it interesting because this does not seem to be the default for that flag (new cmd.exe launch with it disabled, so its technically opt-in). Im really not sure why this behavior would be intended, but you can see that it is the case by running the following expanded example with 0-2 args:

#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
int main()
{
    // Get handle of current stdout
    HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD dwMode = 0;

    if (__argc == 2) {
        printf("Running with system call\n");
        system("Title Something");
    }
    else if (__argc == 3) {
        printf("Running with console flag set\n");
        GetConsoleMode(hOut, &dwMode);
        dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
        SetConsoleMode(hOut, dwMode);
    }

    // Check if the flag is set before printing
    GetConsoleMode(hOut, &dwMode);
    if (dwMode & ENABLE_VIRTUAL_TERMINAL_PROCESSING) {
        printf("Virtual terminal processing is enabled!\n");
    }
    printf("\033[1;31mI am red!\033[0m");
}

I think from that you can also see how to avoid using system() to kludge the virtual console flag and just directly set it. Its unfortunate that this particular behavior is not well documented but I hope this at least helps clear it up.

Answer 2

I think that is a really interesting question and I actually didn't find anything that gives great insights while I was digging. However, there might be some things worth mentioning:

On CPP-Reference they say the following:

Calls the host environment's command processor with the parameter command. Returns an implementation-defined value (usually the value that the invoked program returns).

So it seems that the invocation of the system() function is entirely implementation-defined.

Note how they say "possible output" in the example below:

#include <stdlib.h>
 
int main(void) {
    system("date +%A");
    system("gcc --version");
}

Possible output:

Wednesday
gcc (GCC) 11.2.0
...

On cplusplus.com they give a similar statement:

The effects of invoking a command depend on the system and library implementation, and may cause a program to behave in a non-standard manner or to terminate.

So I conclude:

The system()-function invokes the shell of the operating system, which in turn may have its own initialization routines.

Regarding the initialization of the console environment in Windows, the behavior can be influenced by various factors, including the specific version of Windows and the configuration settings of the command prompt.

I don't think that the initialization process is well-documented in relation to the system function. I hope this helps (even though I'm not entirely sure myself).