httperf buffer overflow detected when rate is high

Question

I'm running httperf 0.9.0 (downloaded from Google Code) on Ubuntu 12.04.1 LTS 64-bit with 2CPU and 4GB RAM. I'm trying to benchmark web servers but encountered the following buffer overflow problem.

Terminal command:

httperf --timeout=5 --client=0/1 --server=localhost --port=9090 --uri=/?value=benchmarks --rate=1200 --send-buffer=4096 --recv-buffer=16384 --num-conns=5000 --num-calls=10

After running for a few seconds, it crashes:

*** buffer overflow detected ***: httperf terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f1f5efa1007]
/lib/x86_64-linux-gnu/libc.so.6(+0x107f00)[0x7f1f5ef9ff00]
/lib/x86_64-linux-gnu/libc.so.6(+0x108fbe)[0x7f1f5efa0fbe]
httperf[0x404054]
httperf[0x404e9f]
httperf[0x406953]
httperf[0x406bd1]
httperf[0x40639f]
httperf[0x4054d5]
httperf[0x40285e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f1f5eeb976d]
httperf[0x4038f1]
======= Memory map: ========
...
...
7f1f5fd74000-7f1f5fd79000 rw-p 00000000 00:00 0 
7f1f5fd91000-7f1f5fd95000 rw-p 00000000 00:00 0 
7f1f5fd95000-7f1f5fd96000 r--p 00022000 08:03 4849686                    /lib/x86_64-linux-gnu/ld-2.15.so
7f1f5fd96000-7f1f5fd98000 rw-p 00023000 08:03 4849686                    /lib/x86_64-linux-gnu/ld-2.15.so
7fff10452000-7fff10473000 rw-p 00000000 00:00 0                          [stack]
7fff1054f000-7fff10550000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

I checked the core dump file with gdb as follows:

(gdb) list
198   event_signal (EV_PERF_SAMPLE, 0, callarg);
199 
200   /* prepare for next sample interval: */
201   perf_sample_start = timer_now ();
202   timer_schedule (perf_sample, regarg, RATE_INTERVAL);
203 }
204 
205 int
206 main (int argc, char **argv)
207 {
(gdb) bt
#0  0x00007f33d4643445 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f33d4646bab in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f33d4680e2e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f33d4716007 in __fortify_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f33d4714f00 in __chk_fail () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f33d4715fbe in __fdelt_warn () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000404054 in set_active (s=<optimized out>, fdset=0x612bc0) at core.c:367
#7  0x0000000000404e9f in core_connect (s=0x17e7100) at core.c:980
#8  0x0000000000406953 in make_conn (arg=...) at conn_rate.c:64
#9  0x0000000000406bd1 in tick (t=<optimized out>, arg=...) at rate.c:94
#10 0x000000000040639f in timer_tick () at timer.c:104
#11 0x00000000004054d5 in core_loop () at core.c:1255
#12 0x000000000040285e in main (argc=11, argv=<optimized out>) at httperf.c:971

I tracked the source code a bit and found that FD_SET seems to be the cause.

Finally, for lower rates (e.g. --rate=100 or --rate=500) httperf works fine. I'm benchmarking different web servers, and the rate causing a crash differs. My rates vary from 100 up to 1200.

For more details, actually I'm trying to repeat the experiments done by Roberto Ostinelli, and I have already tuned TCP settings and applied the patch mentioned in his blog post.

Any idea about what's causing this problem? Thanks!

score 3 · Answer 1 · answered Nov 03 '12 at 04:26

3

You are trying to use an fd that is > 1024. With low loads you don't need/use that many fds. With high loads you need more fds and eventually get to 1024 and that causes the problem.

Even when I increase __FD_SETSIZE I get this problem, so I think there is actually a bug in whatever code is doing the bound check (gcc/llvm?)

answered Nov 03 '12 at 04:26

Me11

31
2

As stated in my question, I have already applied the patch mentioned in the blog post, which is to increase fd related limits in header files. – Xiao Jia Nov 03 '12 at 08:28
So do you have any idea about how to fix this problem? I tried this in Ubuntu 10.04 32 bit and it works fine. Also your content may be suitable for a comment instead of an answer? :-) – Xiao Jia Nov 03 '12 at 08:42

score 3 · Answer 2 · answered Oct 15 '14 at 23:11

Newer versions of glibc do their own checks internally for FD_SET (called from httperf), and those checks fail, causing an abort. Even though httperf was built with a different __FD_SET_SIZE, glibc was still using the original one it was compiled with.

To get around this, I dug up an old version of sys/select.h and bits/select.h, from before __FD_ELT performed checks, and dropped them into httperf's src/ directory (in sys/ and bits). This way, httperf uses the old FD_SET macros that don't perform the checks that cause the abort. I used glibc-2.14.1, but any version that doesn't have bits/select2.h should do the trick.

A friend of mine is collecting this and other fixes to httperf for our own use (and yours!) at https://github.com/klueska/httperf

score 2 · Answer 3 · answered Dec 08 '12 at 16:10

I had a similar crash and in my case replacing select() with epoll() solved it.

I was able to reproduce the problem only in release compilation (I'm using eclipse as devenv, I let eclipse set the compiler options both for debug and release).

Here is the crash:

1354813976/SBNotificationServer terminated
06/12 21:13:54 > ======= Backtrace: =========
06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f4e10f90807]
06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(+0x109700)[0x7f4e10f8f700]
06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(+0x10a7be)[0x7f4e10f907be]
06/12 21:13:54 > 1354813976/SBNotificationServer[0x49db90]
06/12 21:13:54 > 1354813976/SBNotificationServer[0x49de05]
06/12 21:13:54 > 1354813976/SBNotificationServer[0x4a4b07]
06/12 21:13:54 > 1354813976/SBNotificationServer[0x4a5318]
06/12 21:13:54 > 1354813976/SBNotificationServer[0x4a2628]
06/12 21:13:54 > /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f4e10c70e9a]
06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4e10f79cbd]
06/12 21:13:54 > ======= Memory map: ========
06/12 21:13:54 > 00400000-00507000 r-xp 00000000 ca:01 141328         /usr/share/spotbros/SBNotServer/1354813976/SBNotificationServer
06/12 21:13:54 > 00706000-00707000 r--p 00106000 ca:01 141328 /usr/share/spotbros/SBNotServer/1354813976/SBNotificationServer
06/12 21:13:54 > 00707000-00708000 rw-p 00107000 ca:01 141328     /usr/share/spotbros/SBNotServer/1354813976/SBNotificationServer
06/12 21:13:54 > 00708000-0070d000 rw-p 00000000 00:00 0
06/12 21:13:54 > 0120d000-01314000 rw-p 00000000 00:00 0 [heap]
06/12 21:13:54 > 7f49f8000000-7f49f8021000 rw-p 00000000 00:00 0
.......
.......

After this, I was able to generate a core dump and I got:

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./SBNotificationServer ../config.xml'.
Program terminated with signal 6, Aborted.
0 0x00007feb90d67425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) 
(gdb) bt
0 0x00007feb90d67425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
1 0x00007feb90d6ab10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
2 0x00007feb90da539e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
3 0x00007feb90e3b807 in __fortify_fail () from /lib/x86_64-linux-gnu/libc.so.6
4 0x00007feb90e3a700 in __chk_fail () from /lib/x86_64-linux-gnu/libc.so.6
5 0x00007feb90e3b7be in __fdelt_warn () from /lib/x86_64-linux-gnu/libc.so.6
6 0x000000000049e290 in CPhpResponseReader::WaitUntilReadable(int, int&, bool&) ()
7 0x000000000049e505 in CPhpResponseReader::Read(CReallocableBuffer&, int) ()
8 0x00000000004a5207 in CHttpPostBufferInterface::Flush() ()
9 0x00000000004a5a18 in CPhpRequestJob::Execute() ()
10 0x00000000004a2d28 in CThreadPool::Worker(void*) ()
11 0x00007feb90b1be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
12 0x00007feb90e24cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6

MORE DETAILS:

WaitUntilReadable (see the backtrace) is a function that basically uses select() to wait for some data to read from a webserver. I rewrote that function again using epoll() instead.

My program is a server that keeps connections with thousands of clients. The clients make requests to the server, and the server then pass those requests to a webserver, and sends the response back to the clients. For the requests to the webserver, my server has a thread pool. So here comes another important detail: the crash happen if I set the threadpool number of threads to a high number, which meand that a lot of select() calls are being made concurrently (I reproduced it setting it to 1024 threads). Then I launched a client simulator that I made for testing and launched 1000 client threads making requests as fast as they could. This made it crash fast when using select() and compiled in release, but no problem after introducing epoll().

Note that each thread in the pool is making a select() call with 1 fd.

Here is the function I changed to solve this problem:

bool WaitUntilReadableDeprecated(int timeoutMSecs, int& elapsedMSecs, bool& closed)
{
    CTimeLapsus t;
    fd_set fileDescriptorSet;
    struct timeval timeStruct;
    closed = false;

    timeStruct.tv_sec = timeoutMSecs / 1000;
    timeStruct.tv_usec = 0;

    FD_ZERO(&fileDescriptorSet);
    FD_SET(m_socket, &fileDescriptorSet);

    int sel = select(m_socket + 1, &fileDescriptorSet, NULL, NULL, &timeStruct);

    if(sel == 0)
    {
        LogDebug("[CPhpResponseReader::WaitUntilReadable] select() returned 0, no data available");
        elapsedMSecs = t.GetElapsedMilis();
        return false;
    }
    else if(sel == -1)
    {
        if(errno ==  EBADF)
        {
            LogDebug("[CPhpResponseReader::WaitUntilReadable] select() returned -1, errno is EBADF, connection reset by host?");
            closed = true;
            elapsedMSecs = t.GetElapsedMilis();
            return true;
        }

        throw "CPhpResponseReader::WaitUntilReadable select error";
    }

    elapsedMSecs = t.GetElapsedMilis();
    return true;
}

bool WaitUntilReadableEpoll(int timeoutMSecs, int& elapsedMSecs, bool& closed)
{
    CIoPoller poller(8);
    CTimeLapsus t;
    closed = false;

    if(poller.Add(m_socket, EPOLLIN) == -1)
        LogError("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Add(%d, EPOLLIN) failed", m_socket);

    int nfds = poller.Wait(timeoutMSecs);
    if (nfds > 0)
    {
        int theSocket = poller.GetEvents()[0].data.fd;
        uint32_t event = poller.GetEvents()[0].events;
        if(theSocket != m_socket)
        {
            LogError("[CPhpResponseReader::WaitUntilReadableEpoll] socket is different than expected", m_socket);
            elapsedMSecs = t.GetElapsedMilis();
            return false;
        }
        if((event & EPOLLERR) || (event & EPOLLHUP))
        {
            LogWarning("[CPhpResponseReader::WaitUntilReadableEpoll] Disconnected socket %d (event %d)", m_socket, event);
            elapsedMSecs = t.GetElapsedMilis();
            closed = true;
            return false;
        }
        if(event & EPOLLIN)
        {
            // ok
        }
    }
    else if (nfds == -1)
    {
        if(errno ==  EBADF)
        {
            LogWarning("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Wait() returned -1, errno is EBADF, maybe connection reset by host");
            closed = true;
            elapsedMSecs = t.GetElapsedMilis();
            return true;
        }
        LogError("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Wait() failed");
        elapsedMSecs = t.GetElapsedMilis();
        closed = true;
        return false;
    }
    else
    {
        LogDebug("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Wait() returned 0, no data available");
        elapsedMSecs = t.GetElapsedMilis();
        return false;
    }

    elapsedMSecs = t.GetElapsedMilis();
    return true;
}

CIoPoller is just a c++ wrapper for epoll.

Ubuntu version:

Distributor ID: Ubuntu
Description:    Ubuntu 12.04.1 LTS
Release:    12.04

score 0 · Answer 4 · answered Mar 16 '14 at 17:11

0

I meet the same case of buffer overflow:

httperf --hog --wsess=1,3000,1 --rate 1 --burst-length=3000 --max-piped-calls 1 --max-connections=3000 --timeout 20 --server 127.0.0.1 --port 8888 --uri /

look at: https://gist.github.com/diyism/9586262

answered Mar 16 '14 at 17:11

diyism

12,477
5
46
46

httperf buffer overflow detected when rate is high

4 Answers4