Debugging a Ruby segfault

Question

How do I determine if a segfault is due to inconsistent libraries, or a bug in some gem I'm using?

$ uname -a
Linux [redacted] 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ ruby1.9.1 --version
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]

$ CPUPROFILE_OBJECTS=1 CPUPROFILE=/tmp/my_app_profile_objects RUBYOPT="-r`gem1.9.1 which perftools | tail -1`" ruby1.9.1 -e '[].map'
-e:1: [BUG] Segmentation fault
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0004 p:---- s:0009 b:0009 l:000008 d:000008 CFUNC  :map
c:0003 p:0010 s:0006 b:0006 l:0011f8 d:0002f8 EVAL   -e:1
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0011f8 d:0011f8 TOP   

-- Ruby level backtrace information ----------------------------------------
-e:1:in `<main>'
-e:1:in `map'

-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(+0x155b29) [0x7fd39f1dab29] ../sysdeps/x86_64/multiarch/strcpy-ssse3.S:1638
/usr/lib/libruby-1.9.1.so.1.9(+0x57709) [0x7fd39f0dc709] vfscanf.c:1777
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xb7) [0x7fd39f0dd137] vfscanf.c:1796
/usr/lib/libruby-1.9.1.so.1.9(+0xf604f) [0x7fd39f17b04f] wcfuncs.c:49
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7fd39ecfe4c0] ../sysdeps/posix/killpg.c:38
/usr/lib/libruby-1.9.1.so.1.9(rb_check_type+0xd8) [0x7fd39f0de7e8] vfscanf.c:1963
/usr/lib/libruby-1.9.1.so.1.9(rb_data_typed_object_alloc+0xad) [0x7fd39f0f340d] iofdopen.c:51
/usr/lib/libruby-1.9.1.so.1.9(+0x54f9f) [0x7fd39f0d9f9f] vfscanf.c:2014
/usr/lib/libruby-1.9.1.so.1.9(rb_enumeratorize+0x33) [0x7fd39f0dae33] vfscanf.c:2373
/usr/lib/libruby-1.9.1.so.1.9(+0x14e9a1) [0x7fd39f1d39a1] ../sysdeps/x86_64/multiarch/../strcmp.S:293
/usr/lib/libruby-1.9.1.so.1.9(+0x145912) [0x7fd39f1ca912] ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2001
/usr/lib/libruby-1.9.1.so.1.9(+0x14b31d) [0x7fd39f1d031d] ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2032
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xaf) [0x7fd39f1d769f] ../sysdeps/x86_64/multiarch/../strcmp.S:1811
/usr/lib/libruby-1.9.1.so.1.9(+0x5b172) [0x7fd39f0e0172] vfscanf.c:1471
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7fd39f0e0c5d] vfscanf.c:2288
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7fd39f0e280e] psiginfo.c:209
ruby1.9.1() [0x4007db]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fd39ece976d] libc-start.c:226
ruby1.9.1() [0x400809]

-- Other runtime information -----------------------------------------------

* Loaded script: -e

* Loaded features:

    0 enumerator.so
    1 /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
    2 /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
    3 /usr/lib/ruby/1.9.1/rubygems/defaults.rb
    4 /usr/lib/ruby/1.9.1/x86_64-linux/rbconfig.rb
    5 /usr/lib/ruby/1.9.1/rubygems/deprecate.rb
    6 /usr/lib/ruby/1.9.1/rubygems/exceptions.rb
    7 /usr/lib/ruby/1.9.1/rubygems/custom_require.rb
    8 /usr/lib/ruby/1.9.1/rubygems.rb
    9 /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so

* Process memory map:

00400000-00401000 r-xp 00000000 08:07 704429                             /usr/bin/ruby1.9.1
00600000-00601000 r--p 00000000 08:07 704429                             /usr/bin/ruby1.9.1
00601000-00602000 rw-p 00001000 08:07 704429                             /usr/bin/ruby1.9.1
01464000-016e4000 rw-p 00000000 00:00 0                                  [heap]
7fd39c3e1000-7fd39cf53000 rw-p 00000000 00:00 0 
7fd39cf53000-7fd39cf68000 r-xp 00000000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39cf68000-7fd39d167000 ---p 00015000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39d167000-7fd39d168000 r--p 00014000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39d168000-7fd39d169000 rw-p 00015000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39d169000-7fd39d24b000 r-xp 00000000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d24b000-7fd39d44a000 ---p 000e2000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d44a000-7fd39d452000 r--p 000e1000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d452000-7fd39d454000 rw-p 000e9000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d454000-7fd39d469000 rw-p 00000000 00:00 0 
7fd39d469000-7fd39d47d000 r-xp 00000000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d47d000-7fd39d67c000 ---p 00014000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d67c000-7fd39d67d000 r--p 00013000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d67d000-7fd39d67e000 rw-p 00014000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d67e000-7fd39d682000 rw-p 00000000 00:00 0 
7fd39d682000-7fd39d684000 r-xp 00000000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d684000-7fd39d884000 ---p 00002000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d884000-7fd39d885000 r--p 00002000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d885000-7fd39d886000 rw-p 00003000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d886000-7fd39d888000 r-xp 00000000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39d888000-7fd39da87000 ---p 00002000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39da87000-7fd39da88000 r--p 00001000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39da88000-7fd39da89000 rw-p 00002000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39da89000-7fd39e16c000 r--p 00000000 08:07 134744                     /usr/lib/locale/locale-archive
7fd39e16c000-7fd39e265000 r-xp 00000000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e265000-7fd39e464000 ---p 000f9000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e464000-7fd39e465000 r--p 000f8000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e465000-7fd39e466000 rw-p 000f9000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e466000-7fd39e46f000 r-xp 00000000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e46f000-7fd39e66f000 ---p 00009000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e66f000-7fd39e670000 r--p 00009000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e670000-7fd39e671000 rw-p 0000a000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e671000-7fd39e69f000 rw-p 00000000 00:00 0 
7fd39e69f000-7fd39e6a1000 r-xp 00000000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e6a1000-7fd39e8a1000 ---p 00002000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e8a1000-7fd39e8a2000 r--p 00002000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e8a2000-7fd39e8a3000 rw-p 00003000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e8a3000-7fd39e8aa000 r-xp 00000000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39e8aa000-7fd39eaa9000 ---p 00007000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39eaa9000-7fd39eaaa000 r--p 00006000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39eaaa000-7fd39eaab000 rw-p 00007000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39eaab000-7fd39eac3000 r-xp 00000000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39eac3000-7fd39ecc2000 ---p 00018000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39ecc2000-7fd39ecc3000 r--p 00017000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39ecc3000-7fd39ecc4000 rw-p 00018000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39ecc4000-7fd39ecc8000 rw-p 00000000 00:00 0 
7fd39ecc8000-7fd39ee7b000 r-xp 00000000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39ee7b000-7fd39f07a000 ---p 001b3000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39f07a000-7fd39f07e000 r--p 001b2000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39f07e000-7fd39f080000 rw-p 001b6000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39f080000-7fd39f085000 rw-p 00000000 00:00 0 
7fd39f085000-7fd39f0f0000 r-xp 00000000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f0f0000-7fd39f0f3000 rwxp 0006b000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f0f3000-7fd39f275000 r-xp 0006e000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f275000-7fd39f474000 ---p 001f0000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f474000-7fd39f479000 r--p 001ef000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f479000-7fd39f47d000 rw-p 001f4000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f47d000-7fd39f499000 rw-p 00000000 00:00 0 
7fd39f499000-7fd39f4bb000 r-xp 00000000 08:06 5402                       /lib/x86_64-linux-gnu/ld-2.15.so
7fd39f59b000-7fd39f6a1000 rw-p 00000000 00:00 0 
7fd39f6b4000-7fd39f6b5000 rw-p 00000000 00:00 0 
7fd39f6b5000-7fd39f6b6000 ---p 00000000 00:00 0 
7fd39f6b6000-7fd39f6bb000 rw-p 00000000 00:00 0 
7fd39f6bb000-7fd39f6bc000 r--p 00022000 08:06 5402                       /lib/x86_64-linux-gnu/ld-2.15.so
7fd39f6bc000-7fd39f6be000 rw-p 00023000 08:06 5402                       /lib/x86_64-linux-gnu/ld-2.15.so
7fff507f8000-7fff50819000 rw-p 00000000 00:00 0                          [stack]
7fff50941000-7fff50942000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted (core dumped)

I see you have perftools loaded. I couldn't get perftools working with ruby 1.9.3. It segfaulted, like this. I used ruby-prof, which worked a treat for what I needed to do. — d11wtq, Jun 08 '12 at 13:29
I've filed a bug report for this particular scenario: https://github.com/tmm1/perftools.rb/issues/46 — Andrew Grimm, Jun 14 '12 at 01:06
FWIW, I also get a segfault running that on 1.9.3p194 on RVM on Linux Mint 12, though my stack trace is a bit different. — , Jun 14 '12 at 01:22
If you dislike RVM, is rbenv a possibility? (https://github.com/sstephenson/rbenv) — Todd Baur, Jun 20 '12 at 00:02
@ToddBaur Shouldn't that be a comment in wadesworld's answer, rather than this section? — Andrew Grimm, Jun 20 '12 at 00:23

iblue · Answer 1 · 2012-06-18T23:41:10.603

You can find out which part of the software caused a segmentation fault by - and this is probably not a surprise to you - debugging!. As you can see...

Aborted (core dumped)

...there is a core dump.

Just fire up gdb on your executable and the core dump and find out what the parameters to that strcpy (see the first line of your backtrace) were. Then walk backwards trough the stack traces and the code. It seems like debugging symbols have been compiled in, so you are lucky and you can see which function is actually called and what parameters are supplied - at least if the bug did not corrupt the stack.

However, If you are not too familiar with debugging C or assembly code, you probably want to wait for the developers of perftools to debug this issue.

The bug report you mention was by someone with the same name as me. :) — Andrew Grimm, Jun 20 '12 at 01:44

score -2 · Answer 2 · answered Jun 08 '12 at 12:15

-2

That looks like Ruby itself is crashing, so chances are, it's a bug. Ruby 1.9.3 is the latest, so you might try that to see if the bug is fixed.

answered Jun 08 '12 at 12:15

wadesworld

13,535
14
60
93

`$ ruby1.9.1 --version ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]` - In Ubuntu, Ruby 1.9.3 has an executable called "ruby1.9.1". – Andrew Grimm Jun 08 '12 at 12:26
Does it happen with the latest patch level (194)? – d11wtq Jun 08 '12 at 13:26
@d11wtq I haven't tried: I don't want to build Ruby using a non-Debian / non-Ubuntu approach on this machine, and I'm not aware of a Debian / Ubuntu way of installing patchlevel 194. – Andrew Grimm Jun 12 '12 at 01:08
1

What's "non-Ubuntu" approach? Programs installed from source aren't any different, just a little more inconvenient to maintain. However, all Ruby developers should be using [RVM](https://rvm.io/). Use RVM to install and update your Rubies. – wadesworld Jun 12 '12 at 01:13
1

@wadesworld I disagree with RVM quite strongly, though I do think when it's such a fundamental tool you should be prepared to build your own ruby. Locally I use rbenv. In production we just install the ruby we want. – d11wtq Jun 12 '12 at 02:35

Debugging a Ruby segfault

2 Answers2

Linked