embed cmd in win32 application and text not selectable

Question

I write both gui and console program. For console I use color output like \33[0m. For gui I need to write more code, if I switch to another gui library I need to rewrite the code. Some simple library(I'm currently using) doesn't even have api customizing text color. So I try to use cmd as output for all the application.

The problem is that I cannot select text in the console, even if I set my console to QuickEdit mode as default.

The code: (if the cmd doesn't appear, please resize the main window, and then it should appear)

#include <windows.h>

#include <iostream>

LRESULT CALLBACK WndProc(HWND, UINT, WPARAM, LPARAM);

int WINAPI WinMain(HINSTANCE hInstance,
                   HINSTANCE hPrevInstance,
                   LPSTR lpCmdLine,
                   int nShowCmd)
{
    static TCHAR lpszAppName[] = TEXT("HelloWin");
    HWND      hwnd;
    MSG       msg;
    WNDCLASS  wc;

    wc.style         = CS_HREDRAW | CS_VREDRAW;
    wc.lpfnWndProc   = WndProc;
    wc.cbClsExtra    = 0;
    wc.cbWndExtra    = 0;
    wc.hInstance     = hInstance;
    wc.hIcon         = LoadIcon(NULL, IDI_APPLICATION);
    wc.hCursor       = LoadCursor(NULL, IDC_ARROW);
    wc.hbrBackground = (HBRUSH)GetStockObject(WHITE_BRUSH);
    wc.lpszMenuName  = NULL;
    wc.lpszClassName = lpszAppName;

    if (!RegisterClass(&wc))
    {
        MessageBox(NULL, TEXT("This program requires Windows NT!"),
            lpszAppName, MB_ICONERROR);
        return 0;
    }

    hwnd = CreateWindow(lpszAppName,
        TEXT("The Hello Program"),
        WS_OVERLAPPEDWINDOW,
        CW_USEDEFAULT,
        CW_USEDEFAULT,
        CW_USEDEFAULT,
        CW_USEDEFAULT,
        NULL,
        NULL,
        hInstance,
        NULL);

    ShowWindow(hwnd, nShowCmd);
    UpdateWindow(hwnd);

    while (GetMessage(&msg, NULL, 0, 0))
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }

    return msg.wParam;
}

LRESULT CALLBACK WndProc(HWND hwnd, UINT message, WPARAM wParam, LPARAM lParam)
{

    switch (message)
    {
    case WM_CREATE:
        {
            using namespace std;
            AllocConsole();
            freopen("CONOUT$", "w", stdout); // redirect std::cout to console
            cout << "test console" << endl;

            // get console handle
            HWND console = GetConsoleWindow();
            SetParent(console, hwnd);
            SetWindowLong(console, GWL_STYLE, WS_CHILD | WS_VISIBLE);
            // ShowWindow(console, SW_MAXIMIZE);

            DWORD prev_mode;
            GetConsoleMode(console, &prev_mode); 
            SetConsoleMode(console, prev_mode | ENABLE_QUICK_EDIT_MODE);
            cout << "aaaaaaaaaa" << endl;
            cout << "aaaaaaaaaa" << endl;
            cout << "aaaaaaaaaa" << endl;
            cout << "aaaaaaaaaa" << endl;
        }
        return 0;
    case WM_DESTROY:
        PostQuitMessage(0);
        return 0;
    }

    return DefWindowProc(hwnd, message, wParam, lParam);
}

How to make it selectable? And is there any other alternative console application that can be used to embed in my application?(I tried ConEmu but no luck)

Answer 1

SetParent documentation states that if you use this function with another process, you must synchronize the UISTATE for both Windows:

When you change the parent of a window, you should synchronize the UISTATE of both windows. For more information, see WM_CHANGEUISTATE and WM_UPDATEUISTATE.

But you have no access to console's message loop. There are two message loops and Windows has to block some messages. Right away you see problems with focus and painting. The console window doesn't gain focus when you click on it, or it doesn't get painted. Using WS_CLIPCHILDREN will improve painting. To redirect focus, you have to call SetForeground(console) and SetFocus(console) from your own window (this has to be done after WM_CREATE returns, you can handle this in WM_LBUTTONDOWN for example, or with PostMessage) but then you run in to even more problems. Even if you did have access to the other process it wouldn't be easy. Synchronizing threads is hard enough, synchronizing processes would be worse.

See also: Is it legal to have a cross-process parent/child or owner/owned window relationship?
https://blogs.msdn.microsoft.com/oldnewthing/20130412-00/?p=4683

You have easier options. You can slightly modify your code to write to std::ostringstream and paste the stream in to edit control, or redirect cout to an edit control.

The example below uses RichEdit control to support color and font style, loosely based on Bash encoding:

#include <sstream>
#include <string>
#include <iomanip>
#include <Windows.h>
#include <Richedit.h>

class my_stream
{
    HWND hedit;
public:
    std::wostringstream oss;

    HWND create(HWND hwnd, int x, int y, int w, int h, HINSTANCE hinst, int menu_id)
    {
        //create rich edit control
        LoadLibrary(L"Msftedit.dll");
        hedit = CreateWindow(MSFTEDIT_CLASS, 0,
            ES_READONLY | ES_MULTILINE | WS_CHILD | WS_VISIBLE, x, y, w, h, 
            hwnd, HMENU(menu_id), NULL, NULL);

        //default background color
        SendMessage(hedit, EM_SETBKGNDCOLOR, 0, (LPARAM)RGB(0, 0, 0));

        //default text color
        CHARFORMAT cf = { sizeof(cf) };
        cf.dwMask = CFM_COLOR | CFM_FACE | CFM_SIZE;
        cf.yHeight = 220;
        cf.crTextColor = RGB(255, 255, 255);
        //Consolas font is available since Vista
        wcscpy_s(cf.szFaceName, _countof(cf.szFaceName), L"Consolas"); 
        SendMessage(hedit, EM_SETSEL, (WPARAM)0, (LPARAM)-1);
        SendMessage(hedit, EM_SETCHARFORMAT, SCF_SELECTION, (LPARAM)&cf);
        return hedit;
    }

    template<typename T>
    my_stream& operator<<(const T& rhs)
    {
        //write to stream
        oss.str(L"");
        oss << rhs;
        std::wstring s = oss.str();

        if(s.find(L"\033[") == 0)
        {
            bool bold = false;
            if(s.find(L"\033[1") == 0)
            {
                bold = true;
                s[2] = L'0';
            }

            COLORREF color = RGB(255, 255, 255);
            if(s == L"\033[0m") color = RGB(255, 255, 255);
            if(s == L"\033[0;30m") color = RGB(0, 0, 0);//black
            if(s == L"\033[0;31m") color = RGB(255, 0, 0);//red
            if(s == L"\033[0;32m") color = RGB(0, 255, 0);//green
            if(s == L"\033[0;33m") color = RGB(128, 64, 0);//brown
            if(s == L"\033[0;34m") color = RGB(0, 128, 255);//blue
            if(s == L"\033[0;35m") color = RGB(255, 0, 255);//magenta
            if(s == L"\033[0;36m") color = RGB(0, 255, 255);//cyan
            if(s == L"\033[0;37m") color = RGB(192, 192, 192);//light gray

            CHARFORMAT cf = { sizeof(cf) };
            cf.dwMask = CFM_BOLD | CFM_COLOR;
            cf.dwEffects = bold ? CFE_BOLD : 0;
            cf.crTextColor = color;
            SendMessage(hedit, EM_SETSEL, (WPARAM)-1, (LPARAM)-1);
            SendMessage(hedit, EM_SETCHARFORMAT, SCF_SELECTION, (LPARAM)&cf);
        }
        else
        {
            append_to_richedit(s.c_str());
        }

        return *this;
    }

    //this is for std::endl
    my_stream& operator<<(std::wostream& (*func)(std::wostream&))
    {
        oss.str(L"");
        oss << func;
        append_to_richedit(oss.str().c_str());
        return *this;
    }

    void append_to_richedit(const wchar_t *text)
    {
        if(text && wcslen(text))
        {
            SendMessage(hedit, EM_SETSEL, (WPARAM)-1, (LPARAM)-1);
            SendMessage(hedit, EM_REPLACESEL, (WPARAM)FALSE, (LPARAM)text);
        }
    }
};

LRESULT CALLBACK WndProc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam)
{
    static my_stream cout;
    switch(msg)
    {
    case WM_CREATE:
    {
        RECT rc;
        GetClientRect(hwnd, &rc);
        InflateRect(&rc, -10, -10);
        cout.create(hwnd, rc.left, rc.top, rc.right - rc.left, rc.bottom - rc.top,
            ((LPCREATESTRUCT)lparam)->hInstance, 0);

        cout << "\033[0;31m" << "red\n";
        cout << "\033[1;31m" << "bold red\n";
        cout << "\033[0m" << "reset\n";
        cout << "\033[0;32m" << "green\n";
        cout << "\033[0;34m" << std::showbase << std::hex << 17 << std::endl;
        cout << "\033[1m";
        cout << L"bold, unicode ☺ ελληνική\n";
        cout << L"Win10 symbols \n";
        cout.oss.precision(3);
        cout << "numbers " << std::setw(10) << 3.1415 << std::endl;
        break;
    }

    case WM_DESTROY:
        PostQuitMessage(0);
        return 0;
    }

    return DefWindowProc(hwnd, msg, wparam, lparam);
}

int WINAPI wWinMain(HINSTANCE hinst, HINSTANCE, LPTSTR, int)
{
    WNDCLASSEX wcex = { sizeof(WNDCLASSEX) };
    wcex.style = CS_HREDRAW | CS_VREDRAW;
    wcex.lpfnWndProc = WndProc;
    wcex.hInstance = hinst;
    wcex.hCursor = LoadCursor(NULL, IDC_ARROW);
    wcex.hbrBackground = (HBRUSH)(COLOR_BTNFACE + 1);
    wcex.lpszClassName = L"Test";
    RegisterClassEx(&wcex);

    CreateWindow(wcex.lpszClassName, L"Test", WS_VISIBLE | WS_OVERLAPPEDWINDOW, 
        100, 100, 600, 400, 0, 0, hinst, 0);

    MSG msg;
    while(GetMessage(&msg, NULL, 0, 0))
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }

    return (int)msg.wParam;
}

This is a read-only rich edit control, it should support mouse selection and Ctrl+C for copy. You can subclass the richedit control to add menu capabilities.

Answer 2

This can be partially resolved by adding calls to InvalidateRect( console, NULL, TRUE ); and RedrawWindow( console, NULL, NULL, RDW_INVALIDATE );. Try making both calls inside WM_CREATE, WM_MOVE, and WM_SIZE. But call only RedrawWindow inside WM_PAINT as the program seems to freeze up otherwise.

Also call DefWindowProc() after WM_PAINT and other messages. You should only return 0 when sure that no standard windows handling is needed, which is rarely the case unless responding a custom message.

Finally, the Console window is a separate process. Embedding the window of one process in another is inherently dicey. For example, the console normally has a standard window frame, so the text area does not draw at [0,0] of the client rect. It draws to the lower-right of that. Without the window frame, the text area now draws at [0,0] but it's the same size, and so is the surrounding client rect. Which leaves empty unpainted space to the right and bottom of the text area. Setting WS_HSCROLL and WS_VSCROLL on the window, to the show the scrollbars right away, shows blank space between the text area and the scrollbars.

Generally speaking, embedding one process window inside another another is nontrivial. A quick search revealed some related topics here, here. If you want to redirect stdout to an child text window, without using the console, another related topic is here. If you just want the ability to print debug messages, consider using OutputDebugString to print to the Visual Studio output window.