How to display a modal dialog window from another process?

Question

I have a 32-bit MFC application that uses a custom library that would be a nightmare to re-compile into x64. In general the app doesn't really need to run as 64-bit, except in one case -- and that is to render contents to display in a dialog window, which can benefit from a larger addressing space.

So my goal is to "imitate" CDialog::DoModal method but for a dialog in another process.

I built that dialog window as a standalone x64 MFC dialog-based application. It takes a file path as it's input parameter, does all the work internally, and returns simple user selection: OK, Cancel.

So I do the following from my main parent process:

//Error checks omitted for brevity
CString strCmd = L"D:\\C++\\MyDialogBasedApp.exe";

HWND hParWnd = this->GetSafeHwnd();

SHELLEXECUTEINFO sei = {0};
sei.cbSize = sizeof(sei);
sei.fMask = SEE_MASK_FLAG_NO_UI | SEE_MASK_UNICODE | SEE_MASK_NOCLOSEPROCESS;
sei.nShow = SW_SHOW;
sei.lpVerb = _T("open");
sei.lpFile = strCmd.GetBuffer();
sei.hwnd = hParWnd;

BOOL bInitted = SUCCEEDED(::CoInitializeEx(NULL, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE));

ShellExecuteEx(&sei);

DWORD dwProcID = ::GetProcessId(sei.hProcess);

//Try to get main Wnd handle for the child process
HWND hMainChildWnd = NULL;
for(;; ::Sleep(100))
{
    hMainChildWnd = getHwndFromProcID(dwProcID);
    if(hMainChildWnd)
        break;
}

HWND hPrevParWnd = ::SetParent(hMainChildWnd, hParWnd);
if(hPrevParWnd)
{
    //Wait for child process to close
    ::WaitForSingleObject(sei.hProcess, INFINITE);

    //Reset parent back
    ::SetParent(hMainChildWnd, hPrevParWnd);
}

::CloseHandle(sei.hProcess);

if(bInitted)
    ::CoUninitialize();

where getHwndFromProcID is taken from here.

This kinda works, except of the following:

(1) There are two icons on the taskbar: one for my main app, and one for the child app. Is there a way not to show a child icon?

(2) I can switch focus from child window to parent and vice versa. In actual modal dialog window one cannot switch back to parent while child is open. Is there a way to do that?

(3) If I start interacting with the parent, it appears to be "hung up" and the OS will even show it on its title bar.

So I was curious if there's a way to resolve all these?

Answer 1

1. you need pass pointer of own window to child process
2. you need process windows messages, while you wait on child process exit. WaitForSingleObject unacceptably here - need use MsgWaitForMultipleObjectsEx
3. child process must set your window as self owner window at create time - you not need call SetParent

with this all will be worked perfect. in your 32-bit MFC application you need use next code :

BOOL DoExternalModal(HWND hwnd, PCWSTR ApplicationName)
{
    STARTUPINFO si = { sizeof(si) };
    PROCESS_INFORMATION pi;
    WCHAR CommandLine[32];
    swprintf(CommandLine, L"*%p", hwnd);

    if (CreateProcessW(ApplicationName, CommandLine, 0, 0, 0, 0, 0, 0, &si, &pi))
    {
        CloseHandle(pi.hThread);

        MSG msg;

        for (;;)
        {
            switch (MsgWaitForMultipleObjectsEx(1, &pi.hProcess, INFINITE, QS_ALLINPUT, 0))
            {
            case WAIT_OBJECT_0:
                CloseHandle(pi.hProcess);
                return TRUE;
            case WAIT_OBJECT_0 + 1:
                while (PeekMessage(&msg, 0, 0, 0, PM_REMOVE))
                {
                    TranslateMessage(&msg);
                    DispatchMessage(&msg);
                }
                continue;
            default: __debugbreak();
            }
        }
    }

    return FALSE;
}

in MyDialogBasedApp.exe let use MessageBox as demo dialog. we will be use your MFC window as first argument to it.

void ExeEntry()
{
    int ec = -1;

    if (PWSTR CommandLine = GetCommandLine())
    {
        if (CommandLine = wcschr(CommandLine, '*'))
        {
            HWND hwnd = (HWND)(ULONG_PTR)_wcstoi64(CommandLine + 1, &CommandLine, 16);

            if (hwnd && !*CommandLine && IsWindow(hwnd))
            {
                ec = MessageBoxW(hwnd, L"aaa", L"bbb", MB_OK);
            }
        }
    }

    ExitProcess(ec);
}

with this code:

(1) There are only one icon on the taskbar for your main app

(2) You can not switch focus from child window to parent and vice versa. all work as actual modal dialog window

(3) parent not "hung up" because it processing windows messages (MsgWaitForMultipleObjectsEx) - your code is "hung up" because you not do this, but wait in WaitForSingleObject

Answer 2

A modal dialog box does two things that make it "modal":

• The dialog's "owner" is set to the parent window.
• The parent window is disabled.

I played around with this a little bit and while you can do these manually, the easiest way is to just pass the parent window handle to the DialogBox function (or the CDialog constructor in MFC).

Instead of doing all the work after ShellExecuteEx in the parent process, your child process can use FindWindow (or a similar mechanism) to get the parent window handle and use that to display the dialog.

Answer 3

What you are trying to do cannot safely be done. The blog entry Is it legal to have a cross-process parent/child or owner/owned window relationship? explains, that installing a cross-process owner/owned window relationship causes the system to call AttachThreadInput, and - as we all know - AttachThreadInput is like taking two threads and pooling their money into a joint bank account, where both parties need to be present in order to withdraw any money. This creates a very real potential for a dead lock. You can only safely prevent this from happening, if you control both participating threads. Since at least one thread uses a 3rd party application framework (MFC in this case), this is off limits.

Since we have established, that your proposed solution cannot safely be implemented, you need to look into alternatives. One solution might be to delegate the work to a 64-bit process for computation, and have the results passed back to your 32-bit process for display.