Strange behavior in VB6 when calculating a negative number Mod a number which is a power of 2

Question

I was debugging an old piece of code developed in VB6 and found something very strange. It can be demonstrated with the following simple code:

Private Sub Command1_Click()
    Dim a As Integer
    Dim b As Integer
    Dim c As Integer

    a = 0
    Text1.Text = a - 1
    Text2.Text = CStr((a - 1) Mod 4)

    b = 0
    b = b - 1
    Text3.Text = b
    Text4.Text = CStr(b Mod 4)

    c = 0 - 1
    Text5.Text = c
    Text6.Text = CStr(c Mod 4)

End Sub

Here's what the form looked like:

You would think after the button is clicked Text2, Text4 and Text6 should show the same content, which is -1. That is the case when I pressed F5 to run it in the IDE.

This is what it looked like when I ran it in IDE:

The strange thing happened when I made an exe from IDE and ran the exe itself. Text1 and Text6 showed -1 but Text4 showed 3.

This is what it looked like when I ran the exe outside of IDE:

This happens only if the second operand is a power of 2. When I changed the 4's to 5, all the text boxes showed -1.

I tested this on 2 Windows 10 machines and got the same result.

I know VB6 is old and not many people still have it available now for test. I appreciate it if someone can help me understand this.

Thanks.

Seems like a optimization bug. `Text4.Text = CStr(b Mod 4&)` fixes the issue. Or you can use `Long`s. — wqw, Nov 02 '19 at 10:04
Looks indeed like optimization issue. 2 additional possibilities are to 1) is to the the compile option to _Compile to P code_*) instead of _Compile to system code (Native code)_. Or set _Compile to system code (Native code)_ and pick the option _No optimization_. *) I always compiled to P code. Except for number crunching, compiling to native code doesn't do noticeable enhancements. And for number crunching, VB is perhaps not the best tool in town. I remember another weirdness that caused me to settle on P code for everything. — Hel O'Ween, Nov 04 '19 at 10:39
FWIW, I just popped open Calc,exe in Scientific mode, input -1 Mod 4 and the answer was 3. — tcarvin, Nov 05 '19 at 13:08

GSerg · Accepted Answer · 2019-11-08T14:26:20.920

This appears to be a compiler bug. The compiler fails to recognize that the value is signed in case of an Integer (16-bit value), but does respect it in case of a Long.

The code that performs modulo 4 is almost the same in each case and follows the optimized pattern for modulo powers of 2:

Long (b Mod 4&):

or   eax, 0xFFFFFFFF  # eax = 0xFFFFFFFF (which is -1)
and  eax, 0x80000003  # eax = 0x80000003 The modulo op, note it's signed because of the 8

jns  other_code       # Skip the next three lines if the result is non-negative (it isn't here)

dec  eax              # eax = 0x80000002
or   eax, 0xFFFFFFFC  # eax = 0xFFFFFFFE
inc  eax              # eax = 0xFFFFFFFF (which is -1)

other_code:

In the end eax contains 0xFFFFFFFF, which is -1, which is passed for display.

Integer (b Mod 4):

or   eax, 0xFFFFFFFF  # eax = 0xFFFFFFFF, ax = 0xFFFF (which is -1 in both cases)
and  ax,  0x3         # eax = 0xFFFF0003, ax = 0x0003. Should have been "and ax, 0x8003!"

jns  other_code       # Skip the next three lines if the result is non-negative (it incorrectly is)

dec  ax               # Skipped
or   ax,  0xFFFC      # Skipped
inc  ax               # Skipped

other_code:

In the end eax contains 0xFFFF0003 and is then passed to the __vbaStrI2 function that will apparently ignore the two high bytes and only use the 0003.

If and ax, 0x8003 was used instead of the and ax, 0x3, then the skipped lines would trigger and convert the 0xFFFF0003 into 0xFFFFFFFF, which is -1.

With optimizations disabled, the bitwise modulo math is replaced with a straightforward division:

sub   ax, 0x1          # b = b - 1
mov   cx, 0x4          # Prepare division by 4
idiv  cx               # Integer division

As for why the cases for a and c work as expected, this is because the compiler calculates -1 Mod 4 at the compilation stage and hardcodes the result in the executable:

push  0xFFFFFFFF       # Pass hardcoded -1 for display

other_code:

Technically there is nothing that would prevent it from doing the same in the case of b, because it can also prove the value being divided is -1. I cannot tell for sure why it did not do this - maybe it stopped the static analysis one step too soon, or maybe the resulting code for also writing the -1 back to the memory address of b would be less efficient in its opinion.

This explains the different results between Integer and Long. In my original example, all 3 variables are integers, not Long. — elin, Nov 06 '19 at 17:03
@elin For `a` and `c`, the compiler does not have to store the result of `a - 1` and `c - 1` back into `a` and `c`, and it can prove the value before subtraction is 0, so it does not emit code for calculating `Mod` at all and directly pushes `-1` onto the stack as a constant (`push 0xffffffff`). — GSerg, Nov 06 '19 at 20:06

Strange behavior in VB6 when calculating a negative number Mod a number which is a power of 2

1 Answers1