How do I play nice with Cygwin?

Question

I have a C/C++ program, written for Windows, and compiled with Visual Studio. And it's a command-line tool, which means that it could reasonably be run under Cygwin Bash, likely in MinTTY, and some of its users are now doing so.

I'd like to change my program to make it play nicer with Cygwin.

My program currently uses the Windows Console APIs to output pretty colored text and to manage the cursor, and to generally be "interactive" (or "interactive" for a 1980s definition of "interactive"), at least when its output isn't being piped to a file.

To make my program play nicer with Cygwin, I can make it emit ANSI \x1B[... escape codes instead of calling the Console APIs, but the real problem is knowing when to do so.

---

There are four cases, I think:

1. Invoked as an interactive Windows console program. --> (Use Console APIs.)

2. Invoked as a normal Windows console program, but the user is piping its output somewhere. --> (Emit no stylings or interactive calls at all.)

3. Invoked as an interactive program by Bash or under MinTTY. --> (Use ANSI escape codes.)

4. Invoked by Bash, but the user is piping its output somewhere. --> (Emit no stylings or interactive calls at all.)

I can distinguish case 1 by testing whether GetConsoleMode() succeeds or fails for the stdout handle; if it succeeds, I definitely have a Windows Console, and this is the case when I turn on the code to use the Console APIs.

I'd like to treat case 3 differently and use the ANSI codes for it, but unfortunately, cases 2, 3, and 4 seem to be mostly indistinguishable:

• The stdout handle seems to be just an opaque object of FILE_TYPE_PIPE for all three cases.
• I can use the OSTYPE environment variable and at least guess that I'm under Cygwin, which distinguishes case 2 from cases 3 and 4.
• If I could link against cygwin1.dll, I could maybe somehow use its isatty(), but I can't link against that, since many of my users don't (and won't) have Cygwin installed. (And I'm not at all convinced that'd work even if it was possible.)
• MSVC's native _isatty() thinks Cygwin Bash is a pipe, not an interactive shell, because it just uses GetFileType() deep under the hood.

In short, there's no obvious way I can see to separate case 3 from case 4.

Right now, I treat case 3 exactly like cases 2 and 4, and Cygwin users (myself included) get crappy glass teletype interaction instead of a friendly interactive full-screen display, and I'd really like to fix that.

---

So is there any way I can distinguish an interactive Cygwin invocation from other invocations, so my native-Windows program can behave nicely when invoked by Cygwin Bash or other similar shells?

(Important note: There is one solution that's off the table: The program will not be compiled as a native Cygwin program. This is a Windows program, and it's compiled with the Microsoft tool chain, and it will not be compiled with GCC. The goal is to change/update it to behave as nicely under Cygwin as possible — without actually linking against any of the Cygwin DLLs. I can only reasonably distribute one executable file for Windows, not two.)

Answer 1

If I could link against cygwin1.dll, I could maybe somehow use its isatty(), but I can't link against that, since many of my users don't (and won't) have Cygwin installed. (And I'm not at all convinced that'd work even if it was possible.)

Ok you cannot statically link against that DLL, but that doesn't mean that you cannot try to load it dynamically.

You could try to LoadLibrary("cygwin1.dll"). If it fails, then it's not running under cygwin, and you can use native Windows functions to detect console type.

On the other hand, if you can open that library (which means that Cygwin is available in the current system), then you are able to perform a dynamic call to isatty() and use the result to see if the output is redirected.

Edit: it's not that easy to just load the cygwin DLL and call a function, you need to initialize the lib first as shown in this link

An extract:

static BOOL setup_root()
{
    HMODULE hCygwin = NULL;

    // Load the cygwin dll into our application.
    if(!load_cygwin_library(&hCygwin))
        return FALSE;

    // Init the cygwin environment. (Required)
    if(!init_cygwin_library(hCygwin))
        return FALSE;

with both interesting init routines. If init_cygwin_library isn't called, further calls may not work:

static BOOL load_cygwin_library(HMODULE* hCygwin)
{
    if((*hCygwin = GetModuleHandleW(cyglibrary)) == NULL)
        if((*hCygwin = LoadLibraryW(cyglibrary)) == NULL)
            return FALSE;
    return TRUE;
}

static BOOL init_cygwin_library(HMODULE hCygwin)
{
    cygwin_dll_init_fn cygwin_dll_init = NULL;
    if((cygwin_dll_init = (cygwin_dll_init_fn)GetProcAddress(hCygwin,"cygwin_dll_init")) == NULL) {
        FreeLibrary(hCygwin);
        return FALSE;
    }
    cygwin_dll_init();
    return TRUE;
}