GDB breakpoints in trivial program not triggering

Question

I'm working on updating my build of gdb to a later version, and testing using a trivial program is (surprisingly) failing.

#include <stdio.h>

int main()
{
  int a;
  for ( a = 10; a < 20; a = a + 1) {
    printf("value of a: %d\n", a);
    printf("another value of a: %d\n", a);
  }
  return 0;
}

GDB session

[root@localhost ~]# gcc -g main.c
[root@localhost ~]# gdb a.out
GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.out...done.
(gdb) l
1       #include <stdio.h>
2
3       int main()
4       {
5         int a;
6         for ( a = 10; a < 20; a = a + 1) {
7           printf("value of a: %d\n", a);
8           printf("another value of a: %d\n", a);
9         }
10        return 0;
(gdb) b 8
Breakpoint 1 at 0x4004ed: file main.c, line 8.
(gdb) r
Starting program: /root/a.out
value of a: 10
another value of a: 15
value of a: 11
another value of a: 15
value of a: 12
another value of a: 15
value of a: 13
another value of a: 15
value of a: 14
another value of a: 15
value of a: 15
another value of a: 15
value of a: 16
another value of a: 15
value of a: 17
another value of a: 15
value of a: 18
another value of a: 15
value of a: 19
another value of a: 15
[Inferior 1 (process 6083) exited normally]
(gdb)

As you can tell, the breakpoint I place on line 8 never gets triggered. What gives? It's almost like the symbol table is out of sync with the actual stack or something.

Any idea what flags might need to get set? Breakpoints work if I use an older build of gdb from CentOS 6 but unclear why new build would be different.

This GDB was configured as follows:
   configure --host=x86_64-redhat-linux-gnu --target=x86_64-redhat-linux-gnu
             --with-auto-load-dir=$debugdir:$datadir/auto-load
             --with-auto-load-safe-path=$debugdir:$datadir/auto-load
             --with-expat
             --with-gdb-datadir=/usr/share/gdb (relocatable)
             --with-jit-reader-dir=/usr/lib64/gdb (relocatable)
             --without-libunwind-ia64
             --without-lzma
             --with-python=/usr (relocatable)
             --without-guile
             --with-separate-debug-dir=/usr/lib64/debug (relocatable)
             --without-babeltrace

disas main

Dump of assembler code for function main:
   0x00000000004004c8 <+0>:     push   %rbp
   0x00000000004004c9 <+1>:     mov    %rsp,%rbp
   0x00000000004004cc <+4>:     sub    $0x10,%rsp
   0x00000000004004d0 <+8>:     movl   $0xa,-0x4(%rbp)
   0x00000000004004d7 <+15>:    jmp    0x400505 <main+61>
   0x00000000004004d9 <+17>:    mov    -0x4(%rbp),%eax
   0x00000000004004dc <+20>:    mov    %eax,%esi
   0x00000000004004de <+22>:    mov    $0x4005d8,%edi
   0x00000000004004e3 <+27>:    mov    $0x0,%eax
   0x00000000004004e8 <+32>:    callq  0x4003b0 <printf@plt>
   0x00000000004004ed <+37>:    mov    -0x4(%rbp),%eax
   0x00000000004004f0 <+40>:    mov    %eax,%esi
   0x00000000004004f2 <+42>:    mov    $0x4005e8,%edi
   0x00000000004004f7 <+47>:    mov    $0x0,%eax
   0x00000000004004fc <+52>:    callq  0x4003b0 <printf@plt>
   0x0000000000400501 <+57>:    addl   $0x1,-0x4(%rbp)
   0x0000000000400505 <+61>:    cmpl   $0x13,-0x4(%rbp)
   0x0000000000400509 <+65>:    jle    0x4004d9 <main+17>
   0x000000000040050b <+67>:    mov    $0x0,%eax
   0x0000000000400510 <+72>:    leaveq
   0x0000000000400511 <+73>:    retq
End of assembler dump.

objdump

00000000004004c8 <main>:
  4004c8:       55                      push   %rbp
  4004c9:       48 89 e5                mov    %rsp,%rbp
  4004cc:       48 83 ec 10             sub    $0x10,%rsp
  4004d0:       c7 45 fc 0a 00 00 00    movl   $0xa,-0x4(%rbp)
  4004d7:       eb 2c                   jmp    400505 <main+0x3d>
  4004d9:       8b 45 fc                mov    -0x4(%rbp),%eax
  4004dc:       89 c6                   mov    %eax,%esi
  4004de:       bf d8 05 40 00          mov    $0x4005d8,%edi
  4004e3:       b8 00 00 00 00          mov    $0x0,%eax
  4004e8:       e8 c3 fe ff ff          callq  4003b0 <printf@plt>
  4004ed:       8b 45 fc                mov    -0x4(%rbp),%eax
  4004f0:       89 c6                   mov    %eax,%esi
  4004f2:       bf e8 05 40 00          mov    $0x4005e8,%edi
  4004f7:       b8 00 00 00 00          mov    $0x0,%eax
  4004fc:       e8 af fe ff ff          callq  4003b0 <printf@plt>
  400501:       83 45 fc 01             addl   $0x1,-0x4(%rbp)
  400505:       83 7d fc 13             cmpl   $0x13,-0x4(%rbp)
  400509:       7e ce                   jle    4004d9 <main+0x11>
  40050b:       b8 00 00 00 00          mov    $0x0,%eax
  400510:       c9                      leaveq
  400511:       c3                      retq
  400512:       90                      nop
  400513:       90                      nop
  400514:       90                      nop
  400515:       90                      nop
  400516:       90                      nop
  400517:       90                      nop
  400518:       90                      nop
  400519:       90                      nop
  40051a:       90                      nop
  40051b:       90                      nop
  40051c:       90                      nop
  40051d:       90                      nop
  40051e:       90                      nop
  40051f:       90                      nop

GCC version info: gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)

Output outside of gdb:

[root@localhost ~]# ./a.out
value of a: 10
another value of a: 10
value of a: 11
another value of a: 11
value of a: 12
another value of a: 12
value of a: 13
another value of a: 13
value of a: 14
another value of a: 14
value of a: 15
another value of a: 15
value of a: 16
another value of a: 16
value of a: 17
another value of a: 17
value of a: 18
another value of a: 18
value of a: 19
another value of a: 19

Setting a breakpoint at main does not trigger, but output gets (somehow) even weirder:

[root@localhost ~]# gdb a.out
GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.out...done.
(gdb) b main
Breakpoint 1 at 0x4004d0: file main.c, line 6.
(gdb) r
Starting program: /root/a.out
value of a: 0
another value of a: 0
value of a: 1
another value of a: 1
value of a: 2
another value of a: 2
value of a: 3
another value of a: 3
value of a: 4
another value of a: 4
value of a: 5
another value of a: 5
value of a: 6
another value of a: 6
value of a: 7
another value of a: 7
value of a: 8
another value of a: 8
value of a: 9
another value of a: 9
value of a: 10
another value of a: 10
value of a: 11
another value of a: 11
value of a: 12
another value of a: 12
value of a: 13
another value of a: 13
value of a: 14
another value of a: 14
value of a: 15
another value of a: 15
value of a: 16
another value of a: 16
value of a: 17
another value of a: 17
value of a: 18
another value of a: 18
value of a: 19
another value of a: 19
[Inferior 1 (process 2484) exited normally]

Setting a breakpoint at the correct address sort of works, but triggers a SIGILL when continuing:

[root@localhost ~]# gdb a.out
GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.out...done.
(gdb) disas main
Dump of assembler code for function main:
   0x00000000004004c8 <+0>:     push   %rbp
   0x00000000004004c9 <+1>:     mov    %rsp,%rbp
   0x00000000004004cc <+4>:     sub    $0x10,%rsp
   0x00000000004004d0 <+8>:     movl   $0xa,-0x4(%rbp)
   0x00000000004004d7 <+15>:    jmp    0x400505 <main+61>
   0x00000000004004d9 <+17>:    mov    -0x4(%rbp),%eax
   0x00000000004004dc <+20>:    mov    %eax,%esi
   0x00000000004004de <+22>:    mov    $0x4005d8,%edi
   0x00000000004004e3 <+27>:    mov    $0x0,%eax
   0x00000000004004e8 <+32>:    callq  0x4003b0 <printf@plt>
   0x00000000004004ed <+37>:    mov    -0x4(%rbp),%eax
   0x00000000004004f0 <+40>:    mov    %eax,%esi
   0x00000000004004f2 <+42>:    mov    $0x4005e8,%edi
   0x00000000004004f7 <+47>:    mov    $0x0,%eax
   0x00000000004004fc <+52>:    callq  0x4003b0 <printf@plt>
   0x0000000000400501 <+57>:    addl   $0x1,-0x4(%rbp)
   0x0000000000400505 <+61>:    cmpl   $0x13,-0x4(%rbp)
   0x0000000000400509 <+65>:    jle    0x4004d9 <main+17>
   0x000000000040050b <+67>:    mov    $0x0,%eax
   0x0000000000400510 <+72>:    leaveq
   0x0000000000400511 <+73>:    retq
End of assembler dump.
(gdb) b *0x00000000004004fc
Breakpoint 1 at 0x4004fc: file main.c, line 8.
(gdb) r
Starting program: /root/a.out
value of a: 10

Program received signal SIGILL, Illegal instruction.
0x00000000004004fe in main () at main.c:8
8           printf("another value of a: %d\n", a);
(gdb) print {a}
$1 = {10}
(gdb) c
Continuing.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.

Breakpoints at main cause even weirder behavior:

(gdb) break main
Breakpoint 1 at 0x40113f: file main.c, line 6.
(gdb) run
Starting program: /root/gcc731-updated-binutils-a.out
value of a: 0
another value of a: 0
value of a: 1
another value of a: 1
value of a: 2
another value of a: 2
value of a: 3
another value of a: 3
value of a: 4
another value of a: 4
value of a: 5
another value of a: 5
value of a: 6
another value of a: 6
value of a: 7
another value of a: 7
value of a: 8
another value of a: 8
value of a: 9
another value of a: 9
value of a: 10
another value of a: 10
value of a: 11
another value of a: 11
value of a: 12
another value of a: 12
value of a: 13
another value of a: 13
value of a: 14
another value of a: 14
value of a: 15
another value of a: 15
value of a: 16
another value of a: 16
value of a: 17
another value of a: 17
value of a: 18
another value of a: 18
value of a: 19
another value of a: 19
[Inferior 1 (process 1275) exited normally]

I think somehow `gdb` is running an object different from the source file, because `another value of a` should match `value of a`. — jxh, Dec 20 '18 at 23:30
gcc / libc / etc are up to date and from 'official' repository for your Linux version (Centos it seems) ? the non working version of gdb is also the 'official' for repository ? (CentOS probably) — bruno, Dec 20 '18 at 23:48
--without-lzma ? do you have the same configuration with the working old version of gdb ? — bruno, Dec 20 '18 at 23:56
`15` also happens to be the return value of the first `printf` call. — jxh, Dec 21 '18 at 00:05
And you compare gdb's output with the output of `objdump`. See: https://stackoverflow.com/questions/22769246/how-to-disassemble-one-single-function-using-objdump — jxh, Dec 21 '18 at 02:39
@bruno our GCC is just a clean build of 7.3.x that we did when we backported SPECTRE patches back to our kernel. Certainly possible that we messed something up there but I'd be surprised. — javanix, Dec 21 '18 at 14:59
@jxh hah! I didn't even notice that the output was wrong for this program. — javanix, Dec 21 '18 at 15:00
@javanix did you check if you have --without-lzma also in the configuration of the old working gdb ? — bruno, Dec 21 '18 at 15:07
@jxh I'm not much of a C programmer but the objdump / disassembly seem to match. — javanix, Dec 21 '18 at 15:45
@bruno The older (working) build only works for objects compiled with the older gcc 4.4 - it doesn't have --without-lzma but - when we try to breakpoint on objects compiled with new gcc 7.3.1 breakpoints throw a SIGTRAP. — javanix, Dec 21 '18 at 15:46
@javanix is a breakpoint on main (_break main_) works ? if yes doing _stepi_ then _disas main_ to check where you are, if that consistent after some _stepi_ allowing to reach the first printf etc ? For me a _disas_ indicates by => the current instruction, hope the same for you — bruno, Dec 21 '18 at 16:06
@javanix sorry, better to use use _nexti_ rather than _stepi_, to not take the risk to enter in printf and be lost :) When _nexti/stepi_ reaches the beginning of a source line it is written — bruno, Dec 21 '18 at 16:14
@bruno breaking at main doesn't trigger either, but um, a starts incrementing from 0? — javanix, Dec 21 '18 at 16:26
@javanix `b *0x...` to break at a given address, try with some, `b *0x00000000004004ed` for the second printf. — bruno, Dec 21 '18 at 16:29
@javanix btw the `break main` indicates `Breakpoint 1 at 0x4004d0` and this is consistent :( Really I do not understand — bruno, Dec 21 '18 at 16:37
@bruno getting closer - the breakpoint triggers the first time, but continuing from that point triggers a SIGILL. — javanix, Dec 21 '18 at 17:22
(Unrelated: don't debug (or test, or compile for that matter) as root unless you absolutely have too. Bad habit. Bugs can be much more potent when run with admin privs...) — Mat, Dec 26 '18 at 15:52

GDB breakpoints in trivial program not triggering

0 Answers0