0

The docs for ShellExecute state (italicized emphasis mine):

Return value

Type: HINSTANCE

If the function succeeds, it returns a value greater than 32. If the function fails, it returns an error value that indicates the cause of the failure. The return value is cast as an HINSTANCE for backward compatibility with 16-bit Windows applications. It is not a true HINSTANCE, however. It can be cast only to an int and compared to either 32 or the following error codes below.

However, obeying them like so:

if ((int)ShellExecuteW(...) <= 32) ...

Yields warnings when compiled with MSVC 2019 (both 32- and 64-bit):

warning C4311: 'type cast': pointer truncation from 'HINSTANCE' to 'int'
warning C4302: 'type cast': truncation from 'HINSTANCE' to 'int'

Using reinterpret_cast<int> produces similar results.

I can suppress the warnings for that line:

#pragma warning(suppress: 4311 4302)
if ((int)ShellExecuteW(...) <= 32) ...

But I don't generally like doing that unless there are no other options. My question is therefore: Are there other options? Is there some C++ syntax that lets me honor the API documentation without generating warnings?

Jason C
  • 38,729
  • 14
  • 126
  • 182
  • you can use `(DWORD)(DWORD_PTR)ShellExecuteW(...)` cast. (this is same as `PtrToUlong` or `PtrToUint` macros) but better not use this api at all. use `ShellExecuteExW` – RbMm Apr 14 '21 at 19:57
  • @RbMm Does casting to a `DWORD` stay true to the API recommendation? As for `ShellExecuteEx`, there is no reason for me to use that. `ShellExecute` is not deprecated, and I do not need info about the spawned process (this is the only reason given in the API docs for considering `ShellExecuteEx`). – Jason C Apr 14 '21 at 20:10
  • 1
    @JasonC There are other reasons to use `ShellExecuteEx()` than just being able to gather spawned process information. Error handling consistent with other APIs is another reason. – Remy Lebeau Apr 14 '21 at 20:12
  • @RemyLebeau I suppose being able to use `FormatMessage` on error codes *might* be worth the extra typing. Maybe. – Jason C Apr 14 '21 at 20:13
  • 1
    i dont know about " API recommendation", but cast `(DWORD)(DWORD_PTR)` correct. documentation say that return value can be *cast it to an *int* * - so `DWORD` by fact the same. only unsigned – RbMm Apr 14 '21 at 20:17
  • @RbMm I was about to argue with you but double-checked first: I actually didn't know `int` was 32-bit even in 64-bit mode in MSVC ([one of many related posts](https://stackoverflow.com/questions/384502/what-is-the-bit-size-of-long-on-64-bit-windows)) until right this moment. So I guess they *are* the same outside of signedness. TIL. I hardly ever use this compiler and I usually use the explicitly sized types when I care, I guess I just never noticed. – Jason C Apr 14 '21 at 20:20
  • 2
    Doesn't really matter if you use `int` or `DWORD`, the actually values that `ShellExecute()` returns on failure will be in the range `0..32`, which both types represent with exactly the same bits. – Remy Lebeau Apr 14 '21 at 20:20
  • 1
    formally of course `((int)(ULONG_PTR)ShellExecuteW <= 32)` and `(((UINT)(ULONG_PTR)ShellExecuteW <= 32)` - different things. for example if `-1` will be returned - first test will be true and second false. if you want be very strict - you can use `(INT)(INT_PTR)ShellExecuteW` cast or `PtrToInt` macro. but really more easy use `ShellExecuteExW` - this is my recomendation – RbMm Apr 14 '21 at 20:28
  • @RbMm That all makes sense 100%; thank you! – Jason C Apr 14 '21 at 20:30
  • 1
    @JasonC - `PtrToInt` declared in *basetsd.h*, and despited in documentation say about convert to *int* by sense - errors is unsigned here. faster bad documentation. more correct be say about cast to *uint* – RbMm Apr 14 '21 at 20:31

1 Answers1

3

HINSTANCE is a pointer type. The compiler is warning you that the size of the pointer is larger than the size of an int, so bits will be truncated during the cast of HINSTANCE to int. That happens when you are compiling for 64bit, where a pointer is 8 bytes and int is 4 bytes.

In this particular situation, that truncation is fine, as the values will never exceed what an int can hold. But, if you really want to avoid the warnings then simply use intptr_t or INT_PTR or any other similar pointer-sized integer type, eg:

if (reinterpret_cast<intptr_t>(ShellExecuteW(...)) <= 32) {
    ...
}

Otherwise, simply don't use ShellExecute() at all, use ShellExecuteEx() instead. It returns a BOOL for success/failure, and uses GetLastError() to report standard Win32 error codes on failure (most of the error codes returned by ShellExecute() are not standard Win32 error codes), eg:

SHELLEXECUTEINFOW info = {...};

if (!ShellExecuteExW(&info)) {
    DWORD error = GetLastError();
    ...
}
Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • Should I just accept that `intptr_t` / `INT_PTR` are different than the API-documented `int` recommendation and move on with my life, then? – Jason C Apr 14 '21 at 20:11
  • 1
    @JasonC yes. The `ShellExecute()` documentation was written back in a time when 64bit programming didn't even exist yet. If you really want to be thorough, just cast the `intptr_t` to `int` afterwards. – Remy Lebeau Apr 14 '21 at 20:18
  • @JasonC - if value can be casted to *int* - it can be casted to *INT_PTR* too. if you understand how this internal work – RbMm Apr 14 '21 at 20:20
  • All right. I'm convinced. I'll go the `(int)(intptr_t)` style route here. – Jason C Apr 14 '21 at 20:24