Weird behaviour of NOT

Question

I'm experiencing some weird behaviour of boolean variables; the following code prints both "Hello" and "There", meaning result & NOT result both evaluate to True

Dim result As Boolean
result = PostMessage(Application.hWnd, 275, 0, 0)
Debug.Print "Post message: "; result
If result Then Debug.Print "Hello"
If Not result Then Debug.Print "There"

Outputs

Post message: True
Hello
There

According to the docs, PostMessage is declared like this:

BOOL PostMessageA(
  HWND   hWnd,
  UINT   Msg,
  WPARAM wParam,
  LPARAM lParam
);

With the comment:

If the function succeeds, the return value is nonzero.

Here's how I have it in VBA:

Public Declare Function PostMessage Lib "user32" Alias "PostMessageA" ( _
                        ByVal hWnd As LongPtr, _
                        ByVal msg As Long, _
                        ByVal wParam As LongPtr, _
                        ByVal lParam As LongPtr) As Boolean

So what's causing the weird behaviour? How do I get around it?

`PostMessage` doesn't return a VBA `Boolean`. It returns a 32 bit integer. 0 means the function succeeded, non zero means that it failed. Change your `Declare` to correct this mistake. Refer to the documentation: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-postmessagea — David Heffernan, Aug 06 '19 at 11:45
The documentation states "If the function succeeds, the return value is nonzero." which is equivalent to `True` in VBA, though. What is the value of result if you step through the code? — Nacorid, Aug 06 '19 at 11:49
^^Agree with David that the return value is non-Boolean (although I think the interpretation is reversed). I tested with the correction and it worked for me. (BTW it would help a lot with these kinds of things if you'd include a [mcve] in the question. `Handle` and `WM_Timer` are missing...) — Cindy Meister, Aug 06 '19 at 11:50
@CindyMeister Generating a safe handle to use is kind of long and irrelevant; I assume sending a random timer message to the Application's message queue shouldn't have too many unforeseen consequences so I've updated the example:) — Greedo, Aug 06 '19 at 11:54
@Nacorid That's what I thought. The function actually return 1 for success, 0 for fail. I thought VBA's default conversion would put 0 = False, <>0 = True, as is indeed the case when you print. But I believe the problem lies in the fact that VBA represents True as -1 — Greedo, Aug 06 '19 at 11:57
No. It returns non zero for success, not guaranteed to be 1. And yes, I got the logic reversed in my first comment, sorry. — David Heffernan, Aug 06 '19 at 13:23
Here is more context - "logical" operators in VBA work as *bitwise* operators when applied to numbers. The linked answer talks about `And` and `Or`, but it applies to `Not` as well...https://stackoverflow.com/a/8047021/58845 — jtolle, Aug 07 '19 at 00:30

Bathsheba · Answer 1 · 2019-08-06T13:08:26.537

1

Although normally a VBA Boolean holds either True (-1) or False (0), it is possible to insert another value into this type with some trickery.

This is what is happening here: your mis-specified API call is returning an integral type with a value that's neither -1 nor 0.

Since NOT(a) is 0 if and only if a is -1, both Hello and There will be printed if the payload in the Boolean is anything other than -1 or 0.

edited Aug 06 '19 at 13:08

answered Aug 06 '19 at 12:53

Bathsheba

231,907
34
361
483

1

What's wrong with this answer? Is the reason it gives not an accurate explanation for the observed behaviour? – Greedo Aug 11 '19 at 09:54
1

@Greedo: I believe it correct so I'll leave the answer up. Hopefully a downvoter or otherwise might enlighten me - I've answered incorrectly in the past. – Bathsheba Aug 11 '19 at 12:59
@Greedo This answer is correct. That is exactly what happens in VBA when you put something like `1` into a `Boolean` with something like `CopyMemory` or an API call. I cannot find my own answer about it. – GSerg Aug 11 '19 at 19:07

score 0 · Answer 2 · answered Aug 11 '19 at 18:56

Malformed Bools

After a bit of poking and prodding around, I've been able to produce a minimal working example of this issue without the use of any API functions

Private Sub stuff(ByRef into, ByVal val)
    Let into = val
End Sub

Public Sub QuantumBoolTest()

    Dim Schrodinger As Boolean
    Call stuff(Schrodinger, &HFFFE)

    Debug.Print "Schrodinger    :=", Schrodinger & vbCrLf & _
                "NOT Schrodinger:=", (Not Schrodinger)

End Sub

which outputs

Schrodinger    :=           True
NOT Schrodinger:=           True

when executed. As such, it can has been shown that the behaviour you are seeing may be reproduced by stuffing a sixteen bit (in this case signed) integer into the memory address of the bool.

So, in short. your hypothesis is approximately correct.

Interestingly, it seems that the variable now behaves as if it was a signed 16 bit int. This can be seen in a couple of ways, namely, in how it reacts differently to the Sgn function

?sgn(Schrodinger)
-1 

?sgn(Not Schrodinger)
 1 

?sgn(True)
-1 

?sgn(False)
 0

and how it responds to the AND and IMP functions,

?Schrodinger AND NOT Schrodinger
False

?Schrodinger IMP NOT Schrodinger 
True

under which it behaves as if the variable was a signed int,

?&HFFFE AND NOT &HFFFE
0

?&HFFFE IMP NOT &HFFFE
1

and not as if it was a boolean variable.

?TRUE AND NOT TRUE
False

?TRUE IMP FALSE
False

Just a quick note - the functions shown above are immediate window functions, and are all observed by using a breakpoint after the `Call stuff(Schrodinger, &HFFFE)` call. — Taylor Alex Raine, Aug 11 '19 at 18:58
There wasn't really a need for much guesswork. It is documented that `Boolean` is 16 bits, that `False` is `0` (all bits cleared) and that `True` is the bitwise negation of `0` which is `&hFFFF`, or `-1` (all bits set). The `Not` operator expects `True` to be `-1`. — GSerg, Aug 11 '19 at 19:13
@GSerg would you mind sharing that documentation where you got that from (specifically the bit about it being 16 bits)? I had never seen anything that indicated that behavior and would be excited to see any further information on it — Taylor Alex Raine, Aug 13 '19 at 13:35
Um... https://learn.microsoft.com/en-us/office/vba/language/reference/user-interface-help/boolean-data-type? — GSerg, Aug 13 '19 at 15:14

Greedo · Answer 3 · 2019-08-20T08:36:53.220

-1

I guess what's happening is that under the hood, Booleans are saved in VBA as 16-bit integers

False = 0 = &H0000 = 0000 0000 0000 0000

True = -1 = &HFFFF = 1111 1111 1111 1111

And NOT does a bitwise NOT to turn all those 1s into 0s

My function however returns 32-bit 1 which is implicitly cast to 16-bit 1 to store in the Boolean return variable

Result = 1 = &H0001 = 0000 0000 0000 0001

As the value is not 0, it is interpreted by VBA as True (or perhaps more accurately as isn't False since it is not the same value as a normal True)

But bitwise NOT Result gives

NOT Result = -2 = &HFFFE = 1111 1111 1111 1110

Which is also non-zero so is also interpreted as True, giving the behaviour observed.

So to remedy this; either explicitly check the return value by storing it as its native type:

Declare PostMessage(...)  As Long

'...

Dim result as Boolean
result = PostMessage(...) <> 0

Or refrain from using NOT and instead check for False*:

Declare PostMessage(...)  As Boolean

'...

Dim result as Boolean
result = PostMessage(...)

If result = False Then
    '... do blah

*I wouldn't necessarily advise doing this because it makes the code unreadable - I only considered it as an option because it allows you to use intellisense on the return value. But I think it works

edited Aug 20 '19 at 08:36

answered Aug 06 '19 at 12:09

Greedo

4,967
2
30
78

4

Have to downvote for the content starting "or refrain". – David Heffernan Aug 06 '19 at 13:24
@DavidHeffernan Why? Sure it's not especially readable and I would personally choose the first option, but it works doesn't it? Or have I made a mistake? – Greedo Aug 06 '19 at 14:06
2

`PostMessage` returns a 32 bit integer. Compare that against `0`. – David Heffernan Aug 06 '19 at 14:34
@DavidHeffernan Yes I agree that the return type is not a VBA native Boolean. However I think (with care) it can be treated as one. I may be missing something, but I don't think I've written anything *incorrect*, just *inadvisable* - added a note to that effect. I was worried about downcasting errors, but [the docs](https://learn.microsoft.com/en-us/office/vba/language/concepts/getting-started/type-conversion-functions) imply Boolean conversion won't be affected by that. So is there something fundamentally incorrect about this answer (and indeed the other one)? – Greedo Aug 11 '19 at 09:28
I told you what it would take for me to retract my vote. – David Heffernan Aug 11 '19 at 09:40
@DavidHeffernan Oh no, don't worry I'm not too concerned about the votes if they are for stylistic concerns, because that's not what the original question actually asks. I just want to know whether the logic I've detailed here is wrong, the downvotes on the other answer with similar logic would suggest it is? – Greedo Aug 11 '19 at 10:01
2

I find it hard to be motivated to reverse engineer such implementation details. As a general principle with interop it is best to map at binary level as literally as possible, and then wrap for convenience in the target language. – David Heffernan Aug 11 '19 at 10:50
1

Greedo the only difference that I am observing is that while indicating `NOT Result = 65534 = &HFFFE` makes quite a bit of sense, in the context of this question, it would make more sense to treat result as a signed 16 bit integer and read the value as equaling `-2`, as the result of `CLng(NOT Result)` is `-2`. I observe the same values out of `CDbl` and `CInt` – Taylor Alex Raine Aug 11 '19 at 18:13

Weird behaviour of NOT

3 Answers3

Malformed Bools

Linked

Related