How to check ShellExecute command return(HINSTANCE ) code in QT Creator

Question

I am using Shell commands to start a process. My development environment is C++/QT Creator/QT Libraries and Windows APIs.

How can I handle the HINSTANCE return value? if (hInstance < 32) code throwing a compilation error

    437: error: ISO C++ forbids comparison between pointer and integer


    HINSTANCE hInstance = ShellExecute(0, QString("open").toStdWString().c_str(), Path.toStdWString().c_str(), 0, 0, SW_SHOWNORMAL);
   if (hInstance < 32)
      qDebug() << "Error";

ShellExecute() is a screwy 16-bit Windows version 3 left over. You'll have to cast, but do consider ShellExecuteEx() first. — Hans Passant, Jun 05 '12 at 22:07
Or, try CreateProcess(). Besides, why do you need HINSTANCE? — Forgottn, Jun 06 '12 at 07:29
I need to handle the HINSTANCE return code for the error handling. Qprocess has an issue with system which has UAC enabled. — JChan, Jun 06 '12 at 14:07

score 1 · Answer 1 · answered Apr 06 '18 at 13:42

1

If anyone is still struggling with this, here is a solution that avoids warnings:

int retValue = static_cast<int>(reinterpret_cast<uintptr_t>(ShellExecute(<etc>)));

See When is uintptr_t preferred over intptr_t? for more commentary on uintptr_t

answered Apr 06 '18 at 13:42

Adam Gross

56
3

score 0 · Answer 2 · answered Feb 23 '18 at 02:17

You can use reinterpret_cast to cast the return value:

int retValue  = reinterpret_cast<int>(ShellExecute(0, QString("open").toStdWString().c_str(), Path.toStdWString().c_str(), 0, 0, SW_SHOWNORMAL));
   if (retValue  < 32)
      qDebug() << "Error";

while it also can lead to a warning:

warning C4302: 'reinterpret_cast' : truncation from 'HINSTANCE' to 'int'

but at least it works. In MSDN, it also suggest to cast to int.

https://msdn.microsoft.com/en-us/library/windows/desktop/bb762153(v=vs.85).aspx

'If the function succeeds, it returns a value greater than 32. If the function fails, it returns an error value that indicates the cause of the failure. The return value is cast as an HINSTANCE for backward compatibility with 16-bit Windows applications. It is not a true HINSTANCE, however. It can be cast only to an int and compared to either 32 or the following error codes below.'

How to check ShellExecute command return(HINSTANCE ) code in QT Creator

2 Answers2