After much research (and a response from TeamViewer support) I am posting a definitive answer, hope it helpful to others. Code is in Delphi but can be translated easily to C++.
* IF YOU HAVE NO INFO ABOUT THE INTERNALS OF THE PROCESS *
function IsRemoteSupportRunning() : Boolean;
var
hSnapShot, hProcess: THandle;
process: TProcessEntry32;
bFound: Boolean;
begin
bFound := False;
if (cache.dwTeamViewerID = 0) then
begin
// TeamViewer not running...is it running now?
try
hSnapShot := CreateToolHelp32SnapShot(TH32CS_SNAPPROCESS, 0);
process.dwSize := Sizeof(TProcessEntry32);
if (Process32First(hSnapShot, process)) then
bFound := (AnsiCompareText(REMOTE_SUPPORT_EXE, process.szExeFile) = 0);
if (not bFound) then
begin
while (Process32Next(hSnapShot, process)) do
begin
bFound := (AnsiCompareText(REMOTE_SUPPORT_EXE, process.szExeFile) = 0);
if (bFound) then
break;
end;
end;
CloseHandle(hSnapShot);
except
end;
// If found, save the process ID
if (bFound) then
cache.dwTeamViewerID := process.th32ProcessID;
end
else
begin
// In a previous call to this function, TeamViewer was running...
hProcess := OpenProcess(PROCESS_ALL_ACCESS, False, cache.dwTeamViewerID);
if (hProcess > 0) then
begin
// Still running!
bFound := True;
CloseHandle(hProcess);
end
else
begin
// Process is no longer running
cache.dwTeamViewerID := 0;
end;
end;
Result := bFound;
end;
On my machine, this consumes ~1.5ms if TeamViewer.exe is not running. Once running and the PID is known, this drops to ~6.8µs.
* IF YOU HAVE SOME INFO ABOUT, OR HAVE CONTROL OVER, THE PROCESS *
There are a number of possibilities here. For example, the process may create (or you code the process to create) a shared memory object using this code:
CreateFileMapping(HWND($FFFFFFFF), nil, PAGE_READONLY, 0, 32, 'MyProcess');
You can now check for the process running by using this:
var
hMapping: HWND;
hMapping := CreateFileMapping(HWND($FFFFFFFF), nil, PAGE_READONLY, 0, 32, 'MyProcess');
if (hMapping <> 0) then
begin
if (GetLastError() = ERROR_ALREADY_EXISTS) then
bRunning := True;
CloseHandle(hMapping);
end;
Finally, TeamViewer support informed me about a named mutex object which you'll see in the code below. Note that this is specific to TeamViewer, but if you had control over the process, you could create a mutex and use this technique. On my system, this consumes ~2.6µs!
function IsRemoteSupportRunning() : Boolean;
var
hProcess: THandle;
begin
// Use OpenMutex to see if TeamViewer.exe is running...
Result := False;
hProcess := OpenMutex(MUTEX_ALL_ACCESS, False, PChar('TeamViewer_Win32_Instance_Mutex'));
if (hProcess > 0) then
begin
bFound := True;
CloseHandle(hProcess);
end;
end;
TeamViewer informed me of a powerful tool WinObj to investigate objects in the system, https://technet.microsoft.com/de-de/sysinternals/bb896657.aspx. Check out the BaseNamedObjects. Mutexes appear as "Mutant".