How pass a file pointer from c to a call in asm

Question

I'm messing around with nasm, and after doing a hello world with no problem, I though I'd try to do some c integration.

I'm using c to open a file, and I then want to use the pointer returned for the open file to process the text. However, when I call fgetc with the pointer in rdi, I get a "no such file or directory", followed by a segfault.

What am I doing wrong?

int64_t asmFunc(FILE* a, char* b);

int main()
{
   int num;
   FILE *fptr;
   size_t line_buf_size = 0;
   char *ret = malloc(100);
   fptr = fopen("./test.txt","r");

   if(fptr == NULL)
   {
      printf("Error!");   
      exit(1);             
   }

   printf("%ld", asmFunc(fptr, ret));

   return 0;
}

global asmFunc

section .text

extern fgetc

asmFunc:
  call fgetc  ; segfault occurs here.
(...)
  ret

Please give some [MCVE] in your question (including the *real* and full code of `asmFunc`). Without one, it is unclear — Basile Starynkevitch, Apr 27 '19 at 16:13
It'd also be helpful to see the commands you are using to compile and run your program. When I try to build and run your program (after adding the missing headers), I get a dynamic linker error. — Nate Eldredge, Apr 27 '19 at 16:37
Please allow us to Google that for you: [linux x86_64 asm how to call c library function](https://duckduckgo.com/?q=linux+x86_64+asm+how+to+call+c+library+function). [Linux AMD64 call C library functions from copied assembly](https://stackoverflow.com/q/30549375/608639) and [Can't call C standard library function on 64-bit Linux from assembly](https://stackoverflow.com/q/52126328/608639) look promising. — jww, Apr 27 '19 at 17:01

score 3 · Answer 1 · answered Apr 27 '19 at 17:28

The first instruction of asmFunc isn't a call either, but I removed some setup stuff for later operations to make it easier to read.

Well that just defeats the entire purpose of an MCVE. You need to re-run your test after simplifying to make sure it still shows the same problem as your full version. But for this answer, I'll assume your setup didn't clobber the fptr arg in RDI or modify RSP.

asmFunc:
  call fgetc  ; segfault occurs here.

fptr will still be in RDI, where your caller passed it, so that's correct for int fgetc(FILE *fp).

So presumably fgetc is segfaulting because you called it with a misaligned stack. (It was 16-byte aligned before the call that jumped to asmFunc, but you don't have an odd number of pushes or any sub rsp, 8*n). Modern builds of glibc actually do depend on 16-byte alignment for scanf (glibc scanf Segmentation faults when called from a function that doesn't align RSP) so it's easy to imagine that fgetc includes code that also compiles to include a movaps of something on the stack.

Once you fix this bug, you will have the problem that call fgetc destroys your char *ret arg, because your caller passes it in RSI. Arg-passing registers are call-clobbered. What registers are preserved through a linux x86-64 function call

asmFunc:            ; (FILE *fptr,  char *ret)
  push   rsi        ; save ret
  call   fgetc
  pop    rsi
  mov    [rsi], al
  ret

A C compiler would normally save/restore RBX and use mov to save ret there.

asmFunc:            ; (FILE *fptr,  char *ret)
  push   rbx
  mov    rbx, rsi   ; save ret
  call   fgetc
  mov    [rbx], al

  pop    rbx        ; restore rbx
  ret

However, when I call fgetc with the pointer in rdi, I get a "no such file or directory", followed by a segfault.

No idea how you're getting "no such file or directory". Is that from your debugger looking source for glibc functions? If it's part of what your program itself prints, that makes near zero sense, because you do exit(1) correctly when fptr == NULL. And you don't use perror() or anything else that looks up errno codes to generate standard error strings.

I have tested the simplified version. The error I get is: Program received signal SIGSEGV, Segmentation fault. _IO_getc (fp=0x3) at getc.c:37 37 getc.c: No such file or directory. — hellomatey27, Apr 27 '19 at 19:35
@hellomatey27: `getc.c: No such file or directory` is obviously printed by your debugger, like I said. Anyway, if you use your debugger to look at the instruction that faulted, you'll probably see a `movaps` or `movdqa` to or from a stack address that should be aligned but isn't. (Because you violated the stack-alignment requirement in the ABI). — Peter Cordes, Apr 27 '19 at 19:37

Basile Starynkevitch · Answer 2 · 2019-04-27T16:07:40.670

You need to learn and follow the calling conventions documented in Linux x86-64 ABI specification, in particular its §3.2.3 Parameter passing section. So the pointer value fptr is in %rdi, and the pointer value ret is in %rsi and you probably should push a call frame for your asmFunc

Read also the x86 calling conventions wikipage.

If you are able to code the equivalent (even a simplified one) of asmFunc in C in some example.c file, I recommend compiling it with gcc -O -fverbose-asm -Wall -S example.c and looking into the emitted example.s assembler file for inspiration. Most of the time, the first machine instruction of such a function is not a call (but something, called the function prologue, changing the stack pointer %esp and allocating some call frame on the call stack)

For example, on my Linux/Debian/x86-64 with gcc-8

void asmfunc(FILE* fil, char*s) {
   fputc ('\t', fil);
   fputs (s, fil);
   fputc ('\n', fil);
   fflush (fil);
}

is compiled into:

    .text
    .globl  asmfunc
    .type   asmfunc, @function
asmfunc:
.LFB11:
    .cfi_startproc
    pushq   %rbp    #
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    pushq   %rbx    #
    .cfi_def_cfa_offset 24
    .cfi_offset 3, -24
    subq    $8, %rsp    #,
    .cfi_def_cfa_offset 32
    movq    %rdi, %rbx  # fil, fil
    movq    %rsi, %rbp  # s, s
# /tmp/example.c:4:    fputc ('\t', fil);
    movq    %rdi, %rsi  # fil,
    movl    $9, %edi    #,
    call    fputc@PLT   #
# /tmp/example.c:5:    fputs (s, fil);
    movq    %rbx, %rsi  # fil,
    movq    %rbp, %rdi  # s,
    call    fputs@PLT   #
# /tmp/example.c:6:    fputc ('\n', fil);
    movq    %rbx, %rsi  # fil,
    movl    $10, %edi   #,
    call    fputc@PLT   #
# /tmp/example.c:7:    fflush (fil);
    movq    %rbx, %rdi  # fil,
    call    fflush@PLT  #
# /tmp/example.c:8: }
    addq    $8, %rsp    #,
    .cfi_def_cfa_offset 24
    popq    %rbx    #
    .cfi_def_cfa_offset 16
    popq    %rbp    #
    .cfi_def_cfa_offset 8
    ret 
    .cfi_endproc
.LFE11:
    .size   asmfunc, .-asmfunc
    .ident  "GCC: (Debian 8.3.0-6) 8.3.0"

Notice however that in some cases, GCC is capable (e.g. with -O2) of tail-call optimizations and might call some leaf-functions specially.

The way I read them, which is obviously wrong, the first argument needs to be in rdi, which is where the c function places it when it calls my function, right? The first instruction of asmFunc isn't a call either, but I removed some setup stuff for later operations to make it easier to read. — hellomatey27, Apr 27 '19 at 15:58

How pass a file pointer from c to a call in asm

2 Answers2