How do I diagnose this PHP segmentation fault?

Question

I am running a command-line PHP job that is resulting in a Segmentation Fault. This job has worked for a long time, but it processes something that is emailed in. Apparently there's something in this email that's break it, but I have no idea what. If I put the core file in gdb, it's really no help:

$ gdb /usr/local/bin/php core.20381 
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/php...done.
[New Thread 20381]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libmysqlclient_r.so.15...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmysqlclient_r.so.15
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/libmcrypt.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmcrypt.so.4
Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /lib/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpng12.so.0
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libcurl-gnutls.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcurl-gnutls.so.4
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Can't read symbols from system-supplied DSO at 0x7fffb4bfc000: File truncated
Core was generated by `/usr/local/bin/php /src/prod/current/utils/cron.php --run=/utils/proces'.
Program terminated with signal 11, Segmentation fault.
#0  match (
    eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3aa "N", 
    mstart=0x13708600 "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1, maximum-scale=1\">\n\n</head>\n\n<body style=\"padding"..., markptr=0x0, offset_top=4, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17680) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:470
470 {

If I do a 'bt', I just get this "match" statement over and over again. Here are the first few:

#1  0x000000000046ef1e in match (
    eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>, 
    offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17679) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
#2  0x0000000000479789 in match (
    eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>, 
    markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)
    at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623
#3  0x000000000046ef1e in match (
    eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>, 
    offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17677) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803
#4  0x0000000000479789 in match (
    eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>, 
    markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)
    at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623
#5  0x000000000046ef1e in match (
    eptr=0x1370a885 "\t</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>, 
    offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17675) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803

I'm really at a loss here. I've tried running the command in strace, and it seems to be seg-faulting in the PEAR Mail_Queue library. Here's the end of the strace (some identifying information has been removed):

lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/releases/20121025202143/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
open("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0
mmap(NULL, 55723, PROT_READ, MAP_SHARED, 6, 0) = 0x2b49f7bb4000
brk(0x987b000)                          = 0x987b000
brk(0x98bb000)                          = 0x98bb000
munmap(0x2b49f7bb4000, 55723)           = 0
close(6)                                = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
fcntl(6, F_SETFL, O_RDONLY)             = 0
fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
connect(6, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("IP ADDRESS TO DB SERVER REMOVED")}, 16) = -1 EINPROGRESS (Operation now in progress)
fcntl(6, F_SETFL, O_RDWR)               = 0
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}])
setsockopt(6, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(6, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(6, SOL_IP, IP_TOS, [8], 4)   = 0
setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(6, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
poll([{fd=6, events=POLLIN}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}])
read(6, "C\0\0\0\n5.1.37-1ubuntu5.5-log\0Q\257\3\0009"..., 16384) = 71
write(6, "=\0\0\1\205\242\2\0\0\0\0@\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65) = 65
read(6, "\7\0\0\2\0\0\0\2\0\0\0", 16384) = 11
poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(6, "\10\0\0\0\2xxxxx", 12)      = 12
read(6, "\7\0\0\1\0\0\0\2\0\0\0", 16384) = 11
lstat("/src/prod/current/www/application/pear/Mail/mime.php", {st_mode=S_IFREG|0770, st_size=50252, ...}) = 0
lstat("/src/prod/current/www/application/pear/Mail", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/current/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
lstat("/src/prod/current/www/application", {st_mode=S_IFDIR|0776, st_size=4096, ...}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Segmentation fault

If anyone has any ideas or pointers to lines of investigation, I'd truly appreciate it. I'm really stumped on this one.

Answer 1

I usually use xdebug if I encounter a segfault. I prefer to use the following line:

php -d xdebug.auto_trace=ON -d xdebug.trace_output_dir=mytracedir/ myscript.php

With the trace-output under mytracedir/, you can easily figure out the approximate php line that causes the segmentation fault in the script. Then rewrite the line in one statement per line and use error_log() for debugging vars. If you use __destruct() or __toString() it could get a bit more tricky there.

See here for more information.

(If the extension is not loaded, but installed, try to add php -dzend_extension=xdebug.so [...])

Answer 2

The debugger already tells you what happens:

... rdepth=17680) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:470

See pcre.recursion_limit setting:

pcre.recursion_limit - PCRE's recursion limit. Please note that if you set this value to a high number you may consume all the available process stack and eventually crash PHP (due to reaching the stack size limit imposed by the Operating System).

Reduce its value so that it does not crash any longer for the quick fix. You will then see your HTML not fully processed by the regular expression.

Improve the expression to not rely on recursion that much. You probably just want to use a HTML parser.

Answer 3

Try to remove memcache.so and disable opcache from command line php.ini, including opcache section commenting.

Post that run the script with -c path/php.cli.ini and -d memory_limit -1 parameter

Answer 4

Same problem occurred in different situation.

I debug laravel project with phpstorm and when evaluating the expressions php server going down.

Just print Segmentation fault: 11

php version was 7.3 and downgraded to 7.2. And in Phpstorm settings I set php language level to 7.2 also. Problem got solved.

Hope it helps to someone :)

Answer 5

I had this problem. It turned out that commands I was runnnig were for PHP7.1, and on server it was PHP7.3.

Once I ran them like /usr/bin/php71 they were executed without problems.

Hope it helps to someone :)