Get returned char from a C function using inline assembly

Question

I have this function that uses inline assembly that basically calls a C function, gets the returned value, and passes that value as a parameter to another function that returns a character.

void convertText(FILE *arch, FILE *result)
{
    int i = 0;
    int n = arch->size;

    _asm {
    mov esi, 0
whileS:
    cmp esi, n
    jge end

    mov ebx, result
    mov ebx, [ebx]result.information ; Pointer to an array of characters

    push esi ; Push parameters to get5bitsFunc
    push arch ; Push parameters to get5bitsFunc

    call get5bitsFunc
    pop arch ; Restore values
    pop esi ; Restore values

    push eax ; push get5bitsFunc returned value to codify as parameter
    call codify
   mov edi, eax ;  <- HERE move returned value from codify to edi register
    pop eax ; restore eax

    inc esi
    jmp whileS
end:

    }
}

Think of codify as function of the type

unsigned char codify(unsigned char parameter) {
    unsigned char resp;

    // Do something to the parameter
    resp = 'b'; // asign value to resp
    return resp;
}

I have already tested codify and works fine returning the value I want using C code. The problem is that when I run and debug the convertText code in inline assembly in the line I have marked as "-> Here" the value returned in eax is something of the type 3424242 and not 97 or above in the ascii table that is what I need.

How can I get the char value?

An `unsigned char` is only 1 byte while `eax` is a 32-bit (4 byte) register. If `codify()` is only returning 1 byte, then the return value will be stored in `al` (the first byte of `eax`) while leaving the rest of `eax` untouched (which would result in garbage). I would recommend `xor eax, eax` before calling `codify()` so you know that the register is clean before you store the return value in it. — Brendon Boldt, Nov 21 '16 at 22:54
@BrendonBoldt Yeah man, thanks you were right, can you post your comment as an answer so that I can check yours as right? — ravelinx, Nov 21 '16 at 23:06
Why do you `pop eax ; restore eax`? Surely it's just to rebalance the stack, since the next iteration doesn't read it before clobbering it it by calling `get5bitsFunc`. So the instruction is right, but the comment is totally wrong / misleading. Similarly, the POPs after that call are super-weird (and pop into memory is was slower than needed). Also, EDI appears not to be read by anything, so that's weird. I assume you trimmed this down from whatever your real code is. If you're writing this in inline-asm for performance, you're probably doing worse than an optimizing compiler... — Peter Cordes, Nov 21 '16 at 23:24

score 3 · Accepted Answer · edited May 23 '17 at 12:07

The Windows ABI apparently doesn't require functions returning char to zero- or sign-extend the value into EAX, so you need to assume that the bytes above AL hold garbage. (This is the same as in the x86 and x86-64 System V ABI. See also the x86 tag wiki for ABI/calling convention docs).

You can't assume that zeroing EAX before calling codify() is sufficient. It's free to use all of EAX as a scratch register before returning with the char in AL, but garbage in the rest of EAX.

You actually need to movzx esi, al, (or MOVSX), or mov [mem], al or whatever else you want to do to ignore garbage in the high bytes.

score 1 · Answer 2 · answered Nov 21 '16 at 23:15

1

An unsigned char is only 1 byte while eax is a 32-bit (4 byte) register. If codify() is only returning 1 byte, then the return value will be stored in al (the first byte of eax) while leaving the rest of eax untouched (which would result in garbage). I would recommend xor eax, eax before calling codify() so you know that the register is clean before you store the return value in it.

answered Nov 21 '16 at 23:15

Brendon Boldt

235
2
11

Right idea, wrong implementation. codify can use the whole EAX as a temporary before leaving its return value in AL. (I was going to reply to your comment on the question, but then what I was writing started to turn into an answer on its own. Sorry to "steal" your answer after you identified the problem first. I'd upvote this answer if you fixed the recommendation to XOR.) – Peter Cordes Nov 21 '16 at 23:19
Yeah; I see that you are right. It would only work for functions which do not touch `eax` other than to return. – Brendon Boldt Nov 21 '16 at 23:23

Get returned char from a C function using inline assembly

2 Answers2