x86 TEST instruction not working?

Question

I've been banging my head against the wall figuring this out, and this is making no sense to me...

Why does this program enter an infinite loop?!

I thought you could use test to compare two values for equality, as shown here... why doesn't it work?

int main()
{
    __asm
    {
        mov EAX, 1;
        mov EDX, EAX;
        test EAX, EDX;
L:      jne L;
    }
}

@Mysticial: I don't think `mov EAX, 1` can be AT&T syntax, can it? :) — user541686, Mar 03 '13 at 03:54
Oh you're right. Not that it matters anyway. Since `EDX == EAX` regardless. :) — Mysticial, Mar 03 '13 at 03:54
@Mysticial: It's the same exact thing, except that `test EAX, EDX` is turned into `test EDX, EAX`... — user541686, Mar 03 '13 at 03:56
I know, but I'm wondering at the surrounding code. It could be that the compiler is using EAX and EDX for other stuff and because you're corrupting it, it might cause an infinite loop on a larger scope. — Mysticial, Mar 03 '13 at 03:57
@Mysticial: No, it's literally infinite looping from the last jump back to itself... I'm tracing through it instruction by instruction! That's why it's baffling me... — user541686, Mar 03 '13 at 03:57
@PreetSangha: Yeah, it's the same regardless of the register... I just tried `ESI` and `EDI`, same thing. — user541686, Mar 03 '13 at 04:01
The operand label for the jump instruction is pointing at the jump instruction itself? — A. Webb, Mar 03 '13 at 04:04
@A.Webb: Yeah, and it doesn't really matter. You can point it to an earlier instruction if you want to, it'll do the same thing. The real issue is that `jne` doesn't seem to be doing what it says, which is what's confusing me... — user541686, Mar 03 '13 at 04:05
Oh I see what you are thinking, and you should probably be thinking of this same instruction as `JNZ` here. — A. Webb, Mar 03 '13 at 04:13
@A.Webb: Does `JNZ` do what I need? (I'm just trying to loop if `EAX` and `EDX` are unequal.) — user541686, Mar 03 '13 at 04:27
`TEST` is an `AND` that sets flags only, so its not really a test of equality but a test if there is a set bit in common. If you use it with both operands equal to the same register then `TEST` tells you whether there are any bits set, in which case the `JNZ` makes sense. — A. Webb, Mar 03 '13 at 04:37
Let me offer you a hard won lesson: when you are frustrated, its because you have a broken assumption. It took me 10 years to learn this, and I can't use it reliably, because exactly when I need to remember this, I'm frustrated and not thinking logically. In any case, time to examine your assumptions, in this case, about what the instructions do. — Ira Baxter, Mar 03 '13 at 04:51
@IraBaxter: Haha okay, thanks for the advice! Will try to keep it in mind. :) — user541686, Mar 03 '13 at 04:54

score 7 · Accepted Answer · edited Jun 20 '20 at 09:12

Your expectation of what the TEST instruction does is incorrect.

The instruction is used to perform bit tests. You would typically use it to "test" if certain bits are set given a mask. It would be used in conjunction with the JZ (jump if zero) or JNZ (jump if not zero) instructions.

The test involves performing a bitwise-AND on the two operands and sets the appropriate flags (discarding the result). If none of the corresponding bits in the mask are set, then the ZF (zero flag) will be 1 (all bits are zero). If you wanted to test if any were set, you'd use the JNZ instruction. If you wanted to test if none were set, you'd use the JZ instruction.

The JE and JNE are not appropriate for this instruction because they interpret the flags differently.

You are trying to perform an equality check on some variables. You should be using the CMP instruction. You would typically use it to compare values with each other.

The comparison effectively subtracts the operands and only sets the flags (discarding the result). When equal, the difference of the two values is 0 (ZF = 1). When not equal, the difference of the two values is non-zero (ZF = 0). If you wanted to test if they were equal, you'd use the JE (jump if equal) instruction. If you wanted to test if they were not equal, you'd use the JNE (jump if not equal) instruction.

In this case, since you used TEST, the resulting flags would yield ZF = 0 (0x1 & 0x1 = 0x1, non-zero). Since ZF = 0, the JNE instruction would take the branch as you are seeing here.

tl;dr

You need to compare the values using the CMP instruction if you are checking for equality, not TEST them.

int main()
{
    __asm
    {
        mov EAX, 1
        mov EDX, EAX
        cmp EAX, EDX
L:      jne L          ; no more infinite loop
    }
}

+1 awesome, thanks for the explanation. I thought that `test` could be used to test for equality (and I guess I expected it to be faster than `cmp` because it doesn't check for ordering) but I guess not. — user541686, Mar 03 '13 at 04:44
Test can be used to check a register for "equal to zero" (or not) by applying it to the same register as both operands. You can also test for zero by having one operand be constant which is all one bits. You can test for zero in various widths 8,16,32,64 by choosing the appropriate register; that's all pretty messy to explain in a comment. — Ira Baxter, Mar 03 '13 at 04:48

Preet Sangha · Answer 2 · 2013-03-03T04:14:13.153

3

Just reading this (my asm is very rusty) and this

JNE jumps on ZF (Zero Flag) = 0

TEST sets ZF = 0 If bitwise EAX AND EDX results in 1 and 1 if bitwise AND results in 0

If the result of the AND is 0, the ZF is set to 1, otherwise set to 0.

Therefore it jumps as 1 AND 1 results in 0 in ZF.

Seems logical yet counter intuative.

I think @A.Webb is right - it should probably be JNZ if you're using the TEST instruction as you are relying on the behavior of a bitwuse operation to set the zero flag whereas the SUB instruction would set the Zero flag as you need.

edited Mar 03 '13 at 04:14

answered Mar 03 '13 at 04:09

Preet Sangha

64,563
18
145
216

+1 wow, this is so confusing. Why does every piece of code I see use `je` and `jne` then? Don't those work? – user541686 Mar 03 '13 at 04:20
@Mehrdad `jz` and `je` are synonyms, using `jz` makes the intent clearer *to humans*. Disassemblers and assembly listings generated by compilers mostly seem to prefer the `je` variant (I don't know why, and I suppose it doesn't really matter anyway) – harold Mar 03 '13 at 11:21

score 1 · Answer 3 · answered Mar 03 '13 at 04:42

This is pretty simple. You obviously need to know what the instructions do, what processor state they read, and write. When in doubt, get a reference manual. The Intel x86 manuals are easy to find online.

Your specific program:

    mov EAX, 1;

moves the constant 1 to EAX. No other state changes occur.

    mov EDX, EAX;

copies the contexts of EAX into EDX, so it too contains the value 1.

    test EAX, EDX;

test computes the bitwise AND of two registers (did you check the reference manuals?), throws the answer away, and sets condition code bits based on the answer. In your case, the upper 31 bits of each register is zero, and'd produces zeros. The least significant bit is one in both registers; and'd produces a 1. The net effect is that the 32 binary value "one" is generated, and thrown away after the condition code bits are set. There is one condition code bit we care about for this program, and that's the "Z"(ero) bit, which is set if the last condition-code setting operation produced a full zero value. This test produced "one", so the Z bit is reset. I'll let you look up the other condition code bits.

 L:      jne L;

This is a "Jmp on Not Equal", e.g, it jmps if the Z bit is reset. For your program, Z is reset, the jmp occurs. After execution, the processor is at the same insruction, and sees another (the same jmp). The condition code bits aren't changed by a jmp instruction.

So... it goes into an infinite loop.

There are lots of synonyms for various opcodes supported by assemblers. For instance, "JZ" and "JE" are synonyms for the same instruction. Don't let the synonyms confuse.

x86 TEST instruction not working?

Why does this program enter an infinite loop?!

3 Answers3

tl;dr

Linked