x64 NASM Assembly 64bit conditional jump doesn't work

Question

Environment: Ubuntu22.04@WSL on Windows 11, NASM v2.15.05, x64, Intel CPU

When I was debugging my assembly program, I found something strange during my debug. The program was executing a loop. The following code compares between _i(0x404054) variable that is stored in BSS section and the variable number(0x40404c). If _i is less or equal than number, jle _loop should be executed.

(gdb-peda)

   0x4011d2 <_primeTrue+10>    mov    rsi, qword ptr [_i]           <0x404054>
   0x4011da <_primeTrue+18>    call   printf@plt                      <printf@plt>

   0x4011df <_primeTrue+23>    jmp    _primeEnd                      <_primeEnd>
    ↓
   0x4011e2 <_primeEnd>        add    qword ptr [_i], 1             <0x404054>
   0x4011eb <_primeEnd+9>      cmp    qword ptr [0x404054], 0x40404c
 ► 0x4011f7 <_primeEnd+21>   ✔ jle    _loop                      <_loop>
    ↓
  ...(abbreviated)...

At the exact moment when <_primeEnd+21> was running, _i and number's value were the same as following:

gdb-peda$ x/gx 0x404054
0x404054:       0x0000000000000004
gdb-peda$ x/gx 0x40404c
0x40404c:       0x0000000000000003
gdb-peda$

So it means _i values(0x4) is bigger than number value(0x3), so jle _loop shouldn't be executed. However, as you can see on the image below, the program runs jle _loop command even though the condition doesn't seem to be correct.

How is this possible, even though I just obviously confirmed those values Can you let me know what I'm doing wrong? Thanks in advance.

p.s) Context: The full code I'm writing is below. It's just a program that prints all prime numbers until the user input's value. (If I input 100, the program should print every prime number less or equal to 100.)

; Nasm v 2.15.05 (on Linux x64)
extern  printf
extern  scanf

true    EQU     0x1
false   EQU     0x0

section .data
        notificationString              DB      "num: ", 0x0
        notificationStringFormat        DB      "%s", 0x0                               ; %s\0
        userInputFormat                 DB      "%d", 0x0                               ; %d\0
        resultPrintFormat               DB      "%d", 0xA, 0x0                          ; %d\n\0

section .bss
        number                  RESB    0x8
        _i                      RESB    0x8

section .text
        global main

        ; validate if the given number(RSI) is a prime number
        isPrimeNumber:

                mov     r10,  0x2               ; looping variable

                _inspect:
                        mov     rax, rdi        ;rsi
                        cdq
                        mov     rbx, r10
                        div     rbx

                        cmp     rdx, 0x0
                        je      _itIsNotPrime
                        jne     _notDetermined

                        ; None prime number cases go here
                        _itIsNotPrime:
                                mov     rax, false
                                retn

                        _notDetermined:
                                inc     r10
                                mov     r9, r10
                                cmp     r9, rdi         ;rsi
                                je      _itIsPrime      ; Fully inspected and confirmed it is a prime number.
                                jne     _inspect        ; Not fully inspected. Continue checking.

                _itIsPrime:
                        mov rax, true
                        retn

        main:
                ; initialize
                push    rbp
                mov     rbp, rsp

                mov     rdi, notificationString
                mov     rsi, notificationStringFormat
                call    printf

                mov     rdi, userInputFormat
                mov     rsi, number
                call    scanf

                ;mov    rax, 0x2
                ;mov    [_i], rax
                mov     QWORD [_i], 0x2

                _loop:
                        mov     rdi, [_i]
                        call    isPrimeNumber           ; true(0x1)     will return if it is a prime number
                                                        ; false(0x0)    will return if it is not a prime number

                        cmp     rax, true
                        je      _primeTrue
                        jne     _primeFalse
                        _primeTrue:
                                mov     rdi, resultPrintFormat
                                mov     rsi, [_i]
                                call    printf
                                jmp     _primeEnd
                        _primeFalse:
                                nop
                        _primeEnd:

                        ;mov    r10, [_i]
                        ;add    r10, 0x1
                        ;mov    [_i], r10
                        add     QWORD [_i], 0x1

                        cmp     QWORD [_i], number
                        jle     _loop
                _loop_end:

                ; end program
                mov     rsp, rbp
                pop     rbp
                retn

fix) With @Jester's help, I fixed number value reserves 0x8 bytes and _i reserves the same amount. So, there aren't any unexpected values in those.

`print *(long*)0x40404c` or `x/g 0x40404c`. You presumably defined `number` as 32 bit instead of 64. Your edit confirms that. — Jester, Jul 16 '23 at 19:01
@Jester Oh. When I run a command of `x/g 0x40404c`, gdb-peda shows `0x0000000500000005`. It looks like that's not what I intended. — KnightChaser, Jul 16 '23 at 19:02
Yes, your `number` is `resb 4` and also you use `%d` to read it so that's 32 bit. But you then proceed to use it as 64 bit. Either you need to define and read it as 64 bit or make sure you use only 32 bits of it. — Jester, Jul 16 '23 at 19:04
@Jester Thank you so much, I fixed my code after checking your advice, so it looks like the values are not having unintended values anymore. However, the conditional jump still doesn't work as I expected. Could you check my question one more time? I edited my question to the revised version. — KnightChaser, Jul 17 '23 at 03:40
@Jester Solved the problem and I wrote the solution in the answer section. Comment whatever you want after reading if you want. Thanks for your help again. — KnightChaser, Jul 17 '23 at 03:59
You also edited the code in the question to `number RESB 0x8`, removing at least part of the bug that Jester noticed. Apparently you had other bugs as well; the one your answer discusses is a duplicate of [Basic use of immediates vs. square brackets in YASM/NASM x86 assembly](https://stackoverflow.com/q/10362511) — Peter Cordes, Jul 17 '23 at 04:06

score 0 · Accepted Answer · answered Jul 17 '23 at 03:58

I solved the problem. It looks like just using number in the comparison statement didn't reference the exact value in the memory. So, I changed my code to make it refer to the direct memory value by adding the square bracket to do that and comparing them. (number -> [number])

before:

cmp     QWORD [_i], number  ; _i referred correctly, but number isn't.
jle     _loop

After:

mov r10, [_i]
mov r11, [number]
cmp r10, r11
jl  _loop
jnl _loop_end       ; Refer to the full code below. It just means the end of _loop.

The full code I finally finished is:

; Nasm v 2.15.05 (on Linux x64)
extern  printf
extern  scanf

true    EQU     0x1
false   EQU     0x0

section .data
        notificationString              DB      "num: ", 0x0
        notificationStringFormat        DB      "%s", 0x0                               ; %s\0
        userInputFormat                 DB      "%d", 0x0                               ; %d\0
        resultPrintFormat               DB      "%d", 0xA, 0x0                          ; %d\n\0

section .bss
        number                  RESB    0x8
        _i                      RESB    0x8

section .text
        global main

        ; validate if the given number(RSI) is a prime number
        isPrimeNumber:

                mov     r10,  0x2               ; looping variable

                _inspect:
                        mov     rax, rdi        ;rsi
                        cdq
                        mov     rbx, r10
                        div     rbx

                        cmp     rdx, 0x0
                        je      _itIsNotPrime
                        jne     _notDetermined

                        ; None prime number cases go here
                        _itIsNotPrime:
                                mov     rax, false
                                retn

                        _notDetermined:
                                inc     r10
                                mov     r9, r10
                                cmp     r9, rdi         ;rsi
                                je      _itIsPrime      ; Fully inspected and confirmed it is a prime number.
                                jne     _inspect        ; Not fully inspected. Continue checking.

                _itIsPrime:
                        mov rax, true
                        retn

        main:
                ; initialize
                push    rbp
                mov     rbp, rsp

                mov     rdi, notificationString
                mov     rsi, notificationStringFormat
                call    printf

                mov     rdi, userInputFormat
                mov     rsi, number
                call    scanf

                mov     QWORD [_i], 0x2

                _loop:
                        mov     rdi, [_i]
                        call    isPrimeNumber           ; true(0x1)     will return if it is a prime number
                                                        ; false(0x0)    will return if it is not a prime number

                        cmp     rax, true
                        je      _primeTrue
                        jne     _primeFalse
                        _primeTrue:
                                mov     rdi, resultPrintFormat
                                mov     rsi, [_i]
                                call    printf
                                jmp     _primeEnd
                        _primeFalse:
                                nop
                        _primeEnd:

                        add     QWORD [_i], 0x1

                        ; fixed
                        mov     r10, [_i]
                        mov     r11, [number]
                        cmp     r10, r11
                        jle     _loop
                        jmp     _loop_end
                _loop_end:

                ; end program
                mov     rsp, rbp
                pop     rbp
                retn

With that code, now you can get the all prime numbers until your input. (You can also try my code.)

knightchaser@3rdfr13nds:~/assembly/loop$ cat Makefile
prime_numbers_linux_x64: prime_numbers_linux_x64.asm
        nasm -f elf64 -o prime_numbers_linux_x64.o prime_numbers_linux_x64.asm
        gcc -o prime_numbers_linux_x64 prime_numbers_linux_x64.o -no-pie

clean:
        rm -f ./prime_numbers_linux_x64
        rm -f ./prime_numbers_linux_x64.o

Knightchaser@3rdfr13nds:~/assembly/loop$ make
...
knightchaser@3rdfr13nds:~/assembly/loop$ ./prime_numbers_linux_x64
num: 20
3
5
7
11
13
17
19

x64 NASM Assembly 64bit conditional jump doesn't work

1 Answers1