What does ?? in gdb backtrace mean and how to get the actual stack frames?

Question

I was trying to learn how to use gdb on core dumps.

Here is the code:

int main()
{
    return 1/0;
}

This is the gdb output, when I run gdb a.out core:

warning: exec file is newer than core file.
[New LWP 3121]
Core was generated by `./crash'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0  0x00000000004004fc in ?? ()
(gdb) bt
#0  0x00000000004004fc in ?? ()
#1  0x0000000000400500 in ?? ()
#2  0x00007f6ea0945b97 in ?? ()
#3  0x0000000000000000 in ?? ()

What are ?? in the backtrace? How can I resolve them?

Answer 1

Those ?? are usually where the name of the function is displayed. GDB does not know the name of those functions and therefore displays ??.

Now, why is this happening? Depends. GCC compiles including symbols (e.g. function names and similar) by default. Most probably you are working with a stripped version, where symbols have been removed, or just with the wrong file.

As @zwol suggests, the line you see warning: exec file is newer than core file is an indication of the fact that something else is going on that you don't show in your question. You are working on a core dump file generated by the crashed executable, which is outdated.

I would suggest you to re-compile the program from scratch and make sure that you are opening the right file with GDB. First produce a new core dump by crashing the new program, then open it in GDB.

Assuming the following program.c:

int main(void) { return 1/0; }

This should work:

$ rm -f core
$ gcc program.c -o program
$ ./program
Floating point exception (core dumped)

$ gdb program core
Reading symbols from program...(no debugging symbols found)...done.
[New LWP 11896]
Core was generated by `./program'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0  0x000055d24a4cd790 in main ()
(gdb) bt
#0  0x000055d24a4cd790 in main ()
(gdb)

NOTE: if you don't see (core dumped) when running the process that means that a core dump was not generated (which leaves you with the old one). If you are using Bash, try running the command ulimit -c unlimited before crashing the program.

Answer 2

What does ?? in gdb backtrace mean

It means that GDB has no idea to which code the addresses in backtrace: 0x04004fc, 0x0400500, etc. correspond.

and how to get the actual stack frames?

That depends on why this is happening. There are two common scenarios:

1. You are debugging the wrong executable.

   One way this could happen is when you compile with optimization, e.g. gcc -O2 main.c -o crash, let the program dump core, then recompile with debugging (e.g. gcc -g main.c -o crash) and try to debug "old" core dump with "new" executable.

   Don't do that. Instead, compile with optimization and debugging: gcc -O2 -g main.c -o crash.

   P.S. This warning: warning: exec file is newer than core file is intended to warn you precisely about this case.

2. The other common cause is when you obtain a crash on a production system and try to debug it on a development one (given the addresses which you show this is unlikely to have happened here).

   For that case, see this answer.

Answer 3

You did not compile with debug symbols - try adding -g to the compile line