Assembly Return int to C function segfaults

Question

I am finishing up an assembly program that replaces characters in a string with a given replacement character. The assembly code calls C functions and the assembly program itself is called from main in my .c file. However, when trying to finish and return a final int value FROM the assembly program TO C, I get segfaults. My .asm file is as follows:

; File: strrepl.asm
; Implements a C function with the prototype:
;
;   int strrepl(char *str, int c, int (* isinsubset) (int c) ) ;
;
; 
; Result: chars in string are replaced with the replacement character and string is returned.

    SECTION .text
    global  strrepl


_strrepl:   nop
strrepl:
    push    ebp         ; set up stack frame
    mov ebp, esp

    push    esi         ; save registers
    push    ebx
    xor eax, eax
    mov ecx, [ebp + 8]      ;load string (char array) into ecx
    jecxz   end         ;jump if [ecx] is zero
    mov al, [ebp + 12]      ;move the replacement character into esi
    mov edx, [ebp + 16]     ;move function pointer into edx

firstLoop:

    xor eax, eax

    mov edi, [ecx]
    cmp edi, 0
    jz  end

    mov edi, ecx        ; save array


    movzx   eax, byte [ecx]     ;load single byte into eax  
    push    eax         ; parameter for (*isinsubset)
    mov edx, [ebp + 16]         
    call    edx         ; execute (*isinsubset)


    mov ecx, edi        ; restore array
    cmp eax, 0
    jne secondLoop  
    add esp, 4          ; "pop off" the parameter
    mov ebx, eax        ; store return value
    add ecx, 1
    jmp firstLoop

secondLoop:
    mov eax, [ebp+12]
    mov [ecx], al
    mov edx, [ebp+16]
    add esp, 4
    mov ebx, eax
    add ecx, 1
    jmp     firstLoop

end:
    pop ebx         ; restore registers
    pop esi
    mov esp, ebp        ; take down stack frame
    pop ebp
    mov eax, 9
    push    eax         ;test
    ret

and my c file is:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>

//display *((char *) $edi)
// These functions will be implemented in assembly:
//

int strrepl(char *str, int c, int (* isinsubset) (int c) ) ;


int isvowel (int c) {

   if (c == 'a' || c == 'e' || c == 'i' || c == 'o' || c == 'u') 
      return 1 ;

   if (c == 'A' || c == 'E' || c == 'I' || c == 'O' || c == 'U') 
      return 1 ;

   return 0 ;
}

int main(){
    char *str1;
    int r;


    str1 = strdup("ABC 123 779 Hello World") ;
    r = strrepl(str1, '#', &isdigit) ;
    printf("str1 = \"%s\"\n", str1) ;
    printf("%d chararcters were replaced\n", r) ;
    free(str1) ;
    return 0;
}

In my assembly code, you can see in end

mov eax, 9
push    eax

I am simply trying to return the value 9 to the value "r" which is an int in the C file. This is just a test to see if I can return an int back to r in the c file. Eventually I will be returning the number of characters that were replaced back to r. However, I need to figure out why the following code above is segfaulting. Any ideas?

score 1 · Answer 1 · answered May 08 '13 at 08:28

1

mov     eax, 9
push    eax          ; NOT a good idea
ret

That is a big mistake. It's going to return based on the lowest thing on the stack and you've just pushed something on to the stack that's almost certainly not a valid return address.

Most functions return a code by simply placing it into eax (this depends on calling convention of course but that's a pretty common one), there's generally no need to push it on to the stack, and certainly plenty of downside to doing so.

answered May 08 '13 at 08:28

paxdiablo

854,327
234
1,573
1,953

I see. Correct me if I am wrong, but if I wanted to return a value FROM assembly TO c, I would store it in the EAX register, no? – user2357446 May 08 '13 at 08:29
No, not "no" :-) I mean, yes, generally you just shove it into eax and return. – paxdiablo May 08 '13 at 08:30

score 0 · Answer 2 · edited May 23 '17 at 12:03

0

Return values are normally stored in EAX on X86 32 bit machines. So your pushing it on the stack after storing it in EAX is wrong, because the function it is returning to will try to use what is in EAX as a value for IP (instruction pointer)

Ret with no argument pops the return address off of the stack and jumps to it.

source

edited May 23 '17 at 12:03

Community

1
1

answered May 08 '13 at 08:28

Tony The Lion

61,704
67
242
415

I see, so instead of pushing it onto the stack before returning, I should simply change eax to 9, and then return without pushing it? Edit: Just tried this and it worked :). Thanks for this. I forgot that (like the answer below), most functions return a code simply by placing it into eax without the need to push it. – user2357446 May 08 '13 at 08:30
@user2357446 yes that's what you should do :) – Tony The Lion May 08 '13 at 08:31
Well, now my only problem is figuring out how to store a counter to count the number of characters replaced. I've tried simply storing the counter in bl, and then storing bl into [eax] before returning, but that segfaults. I am assuming its because somewhere in the code bl is being changed between function calls – user2357446 May 08 '13 at 08:35
You should ask that in a new question. – Tony The Lion May 08 '13 at 08:36
Asked! Hopefully I am just overlooking something – user2357446 May 08 '13 at 08:47

Assembly Return int to C function segfaults

2 Answers2

Linked