I have a pretty straightforward LAMP stack, with Apache 2.4.23 running a handful of SSL-enabled sites, with PHP 7.1.8 running in a typical mod_php setup. I compiled Apache against OpenSSL 1.0.2j, and everything works great 99% of the time. I've had a handful of segfaults over the past few months, and this morning there was something that just went horribly awry and basically brought the server to its knees. When I rebooted, there was a 2.8 GB core file from Apache, and another 0.9 GB one from the day before. The backtrace seems to both indicate they had issues in the same area. Both have this line:
ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
...followed by a failed assertion from libc, and then the dump:
(gdb) thread apply all bt
Thread 27 (Thread 0x7f99e4ff9700 (LWP 8493)):
#0 0x00007f99fc6693d1 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#1 0x00007f99fc669caf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#2 0x00007f99fc66e5b6 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#3 0x00007f99fc6750c0 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#4 0x0000000000459148 in sig_coredump (sig=6) at mpm_unix.c:959
#5 <signal handler called>
#6 0x00007f99fb1921d7 in raise () from /lib64/libc.so.6
#7 0x00007f99fb1938c8 in abort () from /lib64/libc.so.6
#8 0x00007f99fb18b146 in __assert_fail_base () from /lib64/libc.so.6
#9 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#10 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#11 0x00007f99f86d5df0 in ssl_rand_seed (s=s@entry=0x2605c58, p=0x7f99e8045438, nCtx=nCtx@entry=SSL_RSCTX_CONNECT, prefix=prefix@entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#12 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8045748, r=0x0) at mod_ssl.c:468
#13 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e8045748, csd=csd@entry=0x7f99e80454b0) at connection.c:43
#14 0x000000000046a11a in process_socket (my_thread_num=8, my_child_num=0, cs=0x7f99e80456b8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#15 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#16 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f99e47f8700 (LWP 8494)):
#0 ap_method_number_of (method=0x7f99ac0ce478 "POST") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99ac0ce3f0, r=0x7f99ac0ccee0) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e803f4a8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e803f4a8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e803f4a8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e803f4a8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=9, my_child_num=0, cs=0x7f99e803f418, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f99dcfe9700 (LWP 8509)):
#0 ap_method_number_of (method=0x7f99780c9b10 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99780c9a80, r=0x7f99780c8570) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8053d68) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8053d68) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8053d68) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8053d68) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=24, my_child_num=0, cs=0x7f99e8053cd8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f99e77fe700 (LWP 8488)):
#0 apr_md5_init (context=context@entry=0x7f99e77fdd80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e80495f8, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e8049908, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e8049908, csd=csd@entry=0x7f99e8049670) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=3, my_child_num=0, cs=0x7f99e8049878, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f99e5ffb700 (LWP 8491)):
#0 apr_md5_init (context=context@entry=0x7f99e5ffad80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e804d7b8, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e804dac8, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e804dac8, csd=csd@entry=0x7f99e804d830) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=6, my_child_num=0, cs=0x7f99e804da38, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f99df7ee700 (LWP 8504)):
#0 ap_method_number_of (method=0x7f998411d5d8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f998411d550, r=0x7f998411c040) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e804fba8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e804fba8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e804fba8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e804fba8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=19, my_child_num=0, cs=0x7f99e804fb18, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f99e17f2700 (LWP 8500)):
#0 apr_md5_init (context=context@entry=0x7f99e17f1d80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e8059cf8, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e805a008, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e805a008, csd=csd@entry=0x7f99e8059d70) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=15, my_child_num=0, cs=0x7f99e8059f78, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f99e7fff700 (LWP 8487)):
#0 0x00007f99fb24ee27 in munmap () from /lib64/libc.so.6
#1 0x00007f99fb18b137 in __assert_fail_base () from /lib64/libc.so.6
#2 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#4 0x00007f99f86d5df0 in ssl_rand_seed (s=s@entry=0x258ca08, p=0x7f99e8038ef8, nCtx=nCtx@entry=SSL_RSCTX_CONNECT, prefix=prefix@entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#5 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8039218, r=0x0) at mod_ssl.c:468
#6 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e8039218, csd=csd@entry=0x7f99e8038f80) at connection.c:43
#7 0x000000000046a11a in process_socket (my_thread_num=2, my_child_num=0, cs=0x7f99e8039188, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#8 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#9 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f99e0ff1700 (LWP 8501)):
#0 ap_method_number_of (method=0x7f99a4068468 "POST") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99a40683e0, r=0x7f99a4066ed0) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e803b2f8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e803b2f8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e803b2f8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e803b2f8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=16, my_child_num=0, cs=0x7f99e803b268, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f99e2ff5700 (LWP 8497)):
#0 ap_method_number_of (method=0x7f997acaead8 "POST") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f997acaea50, r=0x7f997acad540) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8037118) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8037118) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8037118) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8037118) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=12, my_child_num=0, cs=0x7f99e8037088, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f99e37f6700 (LWP 8496)):
#0 0x00007f99fb24ee27 in munmap () from /lib64/libc.so.6
#1 0x00007f99fb18b137 in __assert_fail_base () from /lib64/libc.so.6
#2 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#4 0x00007f99f86d5df0 in ssl_rand_seed (s=s@entry=0x258ca08, p=0x7f99e8055b38, nCtx=nCtx@entry=SSL_RSCTX_CONNECT, prefix=prefix@entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#5 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8055e48, r=0x0) at mod_ssl.c:468
#6 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e8055e48, csd=csd@entry=0x7f99e8055bb0) at connection.c:43
#7 0x000000000046a11a in process_socket (my_thread_num=11, my_child_num=0, cs=0x7f99e8055db8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#8 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#9 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f99defed700 (LWP 8505)):
#0 ap_method_number_of (method=0x7f998812ff18 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f998812fe90, r=0x7f998812e980) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8062388) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8062388) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8062388) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8062388) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=20, my_child_num=0, cs=0x7f99e80622f8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f99e3ff7700 (LWP 8495)):
#0 ap_method_number_of (method=0x7f99b005d778 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99b005d6f0, r=0x7f99b005c1e0) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8068628) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8068628) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8068628) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8068628) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=10, my_child_num=0, cs=0x7f99e8068598, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f99e07f0700 (LWP 8502)):
#0 ap_method_number_of (method=0x7f998c004218 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f998c004190, r=0x7f998c002c80) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8047828) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8047828) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8047828) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8047828) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=17, my_child_num=0, cs=0x7f99e8047798, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f99ddfeb700 (LWP 8507)):
#0 ap_method_number_of (method=0x7f99800da758 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99800da6d0, r=0x7f99800d91c0) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8066548) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8066548) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8066548) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8066548) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=22, my_child_num=0, cs=0x7f99e80664b8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f99e27f4700 (LWP 8498)):
#0 ap_method_number_of (method=0x7f999c0e1ba0 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f999c0e1b10, r=0x7f999c0e0600) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e805c0e8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e805c0e8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e805c0e8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e805c0e8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=13, my_child_num=0, cs=0x7f99e805c058, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f99e57fa700 (LWP 8492)):
#0 apr_md5_init (context=context@entry=0x7f99e57f9d80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e8051978, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e8051c88, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e8051c88, csd=csd@entry=0x7f99e80519f0) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=7, my_child_num=0, cs=0x7f99e8051bf8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f99dc7e8700 (LWP 8510)):
#0 0x00007f99fb72d6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000046d3cc in ap_queue_info_wait_for_idler (queue_info=0x257f008, had_to_block=had_to_block@entry=0x7f99dc7e7ebc) at fdqueue.c:180
#2 0x000000000046a523 in get_worker (have_idle_worker_p=0x7f99dc7e7eb0, blocking=1, all_busy=0x7f99dc7e7ebc) at event.c:1328
#3 0x000000000046b2c8 in get_worker (all_busy=0x7f99dc7e7ebc, blocking=1, have_idle_worker_p=0x7f99dc7e7eb0) at event.c:1203
#4 listener_thread (thd=0x2580028, dummy=<optimized out>) at event.c:1762
#5 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f99ed7fe700 (LWP 8485)):
#0 ap_method_number_of (method=0x7f99d40e6ce8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99d40e6c60, r=0x7f99d40e5750) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8043668) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8043668) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8043668) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8043668) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=0, my_child_num=0, cs=0x7f99e80435d8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f99de7ec700 (LWP 8506)):
#0 ap_method_number_of (method=0x7f997c06aa08 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f997c06a980, r=0x7f997c069470) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8064468) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8064468) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8064468) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8064468) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=21, my_child_num=0, cs=0x7f99e80643d8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f99fc871740 (LWP 8483)):
#0 0x00007f99fb73049d in read () from /lib64/libpthread.so.0
#1 0x000000000045a034 in ap_mpm_podx_check (pod=<optimized out>) at mpm_unix.c:535
#2 0x000000000042b3f7 in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at event.c:2368
#3 0x000000000046c2a7 in make_child (s=0x25b8d58, slot=0, bucket=bucket@entry=0) at event.c:2461
#4 0x000000000046cb45 in perform_idle_server_maintenance (num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:2664
#5 server_main_loop (num_buckets=1, remaining_children_to_start=<optimized out>) at event.c:2798
#6 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:2876
#7 0x000000000043271e in ap_run_mpm (pconf=pconf@entry=0x2555138, plog=0x2582378, s=0x25b8d58) at mpm_common.c:94
#8 0x000000000042be71 in main (argc=3, argv=0x7fff71da6a28) at main.c:783
Thread 6 (Thread 0x7f99ecffd700 (LWP 8486)):
#0 ap_method_number_of (method=0x7f99cc006ea8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99cc006e20, r=0x7f99cc005910) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e8057f28) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8057f28) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8057f28) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e8057f28) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=1, my_child_num=0, cs=0x7f99e8057e98, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f99e6ffd700 (LWP 8489)):
#0 0x00007f99fb24ee27 in munmap () from /lib64/libc.so.6
#1 0x00007f99fb18b137 in __assert_fail_base () from /lib64/libc.so.6
#2 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#4 0x00007f99f86d5e88 in ssl_rand_seed (s=s@entry=0x2590578, p=0x7f99e8041278, nCtx=nCtx@entry=SSL_RSCTX_CONNECT, prefix=prefix@entry=0x7f99f86e11dd "") at ssl_engine_rand.c:119
#5 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8041588, r=0x0) at mod_ssl.c:468
#6 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e8041588, csd=csd@entry=0x7f99e80412f0) at connection.c:43
#7 0x000000000046a11a in process_socket (my_thread_num=4, my_child_num=0, cs=0x7f99e80414f8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#8 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#9 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f99dffef700 (LWP 8503)):
#0 ap_method_number_of (method=0x7f99900da7d8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99900da750, r=0x7f99900d9240) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e804b9e8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e804b9e8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e804b9e8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e804b9e8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=18, my_child_num=0, cs=0x7f99e804b958, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f99e67fc700 (LWP 8490)):
#0 apr_md5_init (context=context@entry=0x7f99e67fbd80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e805deb8, buf=0x2585780 "(DOMAIN NAME #2 REMOVED):443", length=20) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e805e1c8, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e805e1c8, csd=csd@entry=0x7f99e805df30) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=5, my_child_num=0, cs=0x7f99e805e138, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f99e1ff3700 (LWP 8499)):
#0 ap_method_number_of (method=0x7f99a0004218 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99a0004190, r=0x7f99a0002c80) at protocol.c:634
#2 ap_read_request (conn=conn@entry=0x7f99e80602a8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e80602a8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e80602a8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c@entry=0x7f99e80602a8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=14, my_child_num=0, cs=0x7f99e8060218, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f99dd7ea700 (LWP 8508)):
#0 0x00007f99fb1921d7 in raise () from /lib64/libc.so.6
#1 0x00007f99fb1938c8 in abort () from /lib64/libc.so.6
#2 0x00007f99fb18b146 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#5 0x00007f99f86d5df0 in ssl_rand_seed (s=s@entry=0x2605c58, p=0x7f99e803d0b8, nCtx=nCtx@entry=SSL_RSCTX_CONNECT, prefix=prefix@entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#6 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e803d3c8, r=0x0) at mod_ssl.c:468
#7 0x0000000000456edb in ap_run_pre_connection (c=c@entry=0x7f99e803d3c8, csd=csd@entry=0x7f99e803d130) at connection.c:43
#8 0x000000000046a11a in process_socket (my_thread_num=23, my_child_num=0, cs=0x7f99e803d338, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#9 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#10 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f99fb25473d in clone () from /lib64/libc.so.6
However, I'm no expert at reading core dumps - I've read through a few posts like How to analyze a program's core dump file with gdb? and https://httpd.apache.org/dev/debugging.html, but I'm not quite sure where to go next.
Should I be assuming that there's something wrong in my OpenSSL library and I should recompile it? Or am I missing something else? All feedback welcome!
