1

When trying to debug (after compiling and linking) assembly 86-64x program called hello_world, I got a gdb error "not in executable format: file format not recognized".

ubuntu@ubuntu:~$ gdb hello_world
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/home/ubuntu/hello_world": not in executable format: File format not recognized

I use Ubuntu 64x OS and gdb 8.1.0 64x.

I have looked for other answers, but couldn't understand what to do or the solution was for mac OS.

When running 

`ubuntu@ubuntu:~$ file hello_world

I got 

hello_world: ASCII text

After looking this answer I understood that gdb doesn't know what to do with this file, but I didn't figure out how to change the file's format.

My hello_world program:

global _start

section .text

 _start:
  mov rax,1
  mov rdi,1
  mov rsi,message
  mov rdx,13

  syscall

  mov rax,60
  xor rdi,rdi

  syscall

  section .data
  message: db "Hello, World",10

I have compiled and linked using the next commands:

   ubuntu@ubuntu:~$ nasm -felf64 hello_world
   ubuntu@ubuntu:~$ ld hello_world.o
Sharon1111
  • 11
  • 1
  • 3
  • 1
    What commands did you use to build this file, from what source? What do the first 10 lines look like? `head hello_world`. If it's *really* ASCII text, post it in your question so we can tell you what kind of file it actually is. Actual ELF executables are never mistaken for ASCII text- they start with a non-ASCII "magic number" in binary. – Peter Cordes Mar 19 '19 at 15:10
  • As Peter asked, how did you assemble and link? Are you loading the correct file (the output from the linker)? – Jester Mar 19 '19 at 15:16
  • 2
    So, `hello_world` is your source. You should really name it `hello_world.asm` as per convention. `ld` produces output named `a.out` by default so load that into `gdb`. You can specify other name with the `-o` option. Make sure not to overwrite your input though :) – Jester Mar 19 '19 at 15:25

1 Answers1

7
   ubuntu@ubuntu:~$ nasm -felf64 hello_world
   ubuntu@ubuntu:~$ ld hello_world.o

hello_world is your source file; it's what you ran NASM on. Normally you'd name a NASM source file hello_world.asm, like a C hello_world.c.

The default output file for ld is a.out, so your command created an executable called a.out. If you want to create an executable called hello_world, you need to use
ld -o hello_world hello_world.o.

(Which would overwrite your source unless you first renamed it to .asm. This is why the convention is to use an extension on source files.)

You could have gotten a hint by running ls -lcrt to sort a directory listing by inode-change time. You'll see a.out at the bottom, after hello_world.o, which will remind you that ld created that instead of hello_world.

fuz
  • 88,405
  • 25
  • 200
  • 352
Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
  • @fuz: GNU binutils `ld` accepts `-o output` before or after the input filenames. The man page only documents putting options before inputs, but in practice it works either way. I'll let your edit stand because it does match the documentation. – Peter Cordes Mar 19 '19 at 17:32
  • 1
    Not all toolchains work this way and it's not a good idea to popagate this bad habit. The [Utility Syntax Guidelines](http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html) of the POSIX standard specify that options come before operands; the GNU project is the major player not sticking to this convention. – fuz Mar 19 '19 at 18:06