4

I am calling memcmp from fasm (version 1.71.51) code, and I obtain strange results.

It seems like memcmp only compares characters that are at odd position. Code:

format ELF64

section '.text' executable

    public _start

    extrn memcmp
    extrn exit

_start:
    push    rbp     ; Align the stack to 16 bytes

    mov     rdi, str1
    mov     rsi, str2
    mov     rdx, BUFF_LEN
    call    memcmp

    mov     rdi, rax
    call    exit
    pop     rbp


section '.data' writeable

str1                db '1509487271'
BUFF_LEN = $ - str1
str2                db '1509487273'

I am running Ubuntu:

$ uname -a
Linux freedom 4.13.0-43-generic #48~16.04.1-Ubuntu SMP Thu May 17 12:56:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

To assemble and run the code above, I use the following commands:

$ fasm strange_memcmp.asm && ld -dynamic-linker /lib64/ld-linux-x86-64.so.2 -lc -melf_x86_64 strange_memcmp.o -o strange_memcmp.out && ./strange_memcmp.out 
flat assembler  version 1.71.51  (16384 kilobytes memory)
3 passes, 803 bytes.
$ echo $?
0

I expect that memcmp will return a value that is not 0, because the buffers are different.

I tried changing the two buffers. If I set:

str1 = '2509487271'
str2 = '1509487273'

I obtain a return value of 1.

If I set:

str1 = '1409487271'
str2 = '1509487273'

I obtain a return value of 0. It seems like memcmp cares only about characters at even positions: 0, 2, 4, ... etc.

I have a strange feeling that I am calling memcmp incorrectly. This might have something to do with Linux x64 abi, but I can't find out what could be the problem. Any ideas are appreciated!

EDIT: According to Raymond's answer, the return value from the process is anded with a mask.

To fix the code above, I added the two instructions:

neg     rax
sbb     rax, rax

right after the call to memcmp.

real
  • 639
  • 1
  • 6
  • 15
  • Your call looks correct. I am not sure what the problem is, to be honest. – fuz Jun 16 '18 at 16:43
  • Possible duplicate of [Why does the program return with an exit code other than I specified?](https://stackoverflow.com/questions/34111084/why-does-the-program-return-with-an-exit-code-other-than-i-specified) – Raymond Chen Jun 16 '18 at 17:35
  • 1
    BTW, the stack is aligned by 16 on entry to `_start`. push misaligns it. It's *not* a function; so it's different from the usual aligned-before-the-`call` situation. – Peter Cordes Jun 16 '18 at 17:39
  • 1
    What glibc version are you using? What is the *actual* memcmp return value? (`ltrace ./a.out` is one easy way to check, or use `strace` to look at the arg being passed to `sys_exit_group`). On my Arch Linux system, I ported your code to NASM. With `1409487271` vs. `1509487273`, I get `memcmp` returning `-1`, so my process exit status is 255. glibc `memcmp` returns by extending the bytes to `int` and subtracting them, so the low byte is always non-zero if the `int` return value is non-zero. – Peter Cordes Jun 16 '18 at 17:48
  • @Peter Cordes: Thanks for the _start advice. I want to be sure I understand: can I call directly to libc functions from _start, without having to align anything? I thought I have to push something of size 8 bytes. – real Jun 16 '18 at 17:50
  • 2 things: many `libc` functions don't actually depend on stack alignment, which is why your program doesn't fail. The x86-64 System V ABI says that RSP must be 16-byte aligned before a `call` (which pushes a return address). So at the start of a function, one `push` sets you up for another `call`. But `_start` is not a function, and is entered with the stack 16-byte aligned, rather than with `rsp-8` 16-byte aligned. (The ABI also guarantees this. Other than the struct-passing rules, it's very readable: https://github.com/hjl-tools/x86-psABI/wiki/X86-psABI.) – Peter Cordes Jun 16 '18 at 18:08
  • See also https://stackoverflow.com/tags/x86/info – Peter Cordes Jun 16 '18 at 18:08
  • Anyway, I can't repro it on my system, but a different `memcmp` implementation is allowed to behave differently. glibc does have several to choose from (optimized for different CPU feature sets, like AVX2), so maybe it is doing dword subtraction or something instead of sign-extending a byte. And Ubuntu 16.04's gcc is a couple years older than my Arch system. – Peter Cordes Jun 16 '18 at 18:18

1 Answers1

12

You are passing the 64-bit value returned from memcmp directly to exit, but the value is ANDed with 0xFF before being returned to the parent. memcmp is probably returning a nonzero value like 512, but the system truncates it to 0.

CherryDT
  • 25,571
  • 5
  • 49
  • 74
Raymond Chen
  • 44,448
  • 11
  • 96
  • 135