Linux - segmentation fault only sometimes - how to debug

Question

I have a Linux program, that from time to time ends with a segmentation fault. The program is running periodically every hour, but the segmentation fault occurs only sometimes.

I have a problem to debug this, because if I run the program again with the same input, no error is reported and all is OK.

Is there a way, how to "report" in which part of the code error occured or what caused the problem?

Run the program under `gdb`? Get a core dump? What have you tried? — David Schwartz, Dec 20 '16 at 09:18
Try to log variable states. Also, you could try to run it under `valgrind`. — Yaeger, Dec 20 '16 at 09:19
@gowrath I have tried print statements, but it is a problem because error occured randomly. So I try to repair something, it runs fine afterwards, but the next day, crash again — Martin Perry, Dec 20 '16 at 09:21
I am not talking about memory leaks @MartinPerry. If you compile your program with the "-g" option and run it through valgrind, it will give you a report about where (on which line number) a segmentation file occurred. — Yaeger, Dec 20 '16 at 09:29

score 4 · Answer 1 · answered Dec 20 '16 at 09:19

4

The usual way is to have the crashing program generate a corefile and analyze this after the crash. Make sure, that:

the maximum corefile-size is big enough (i.e. unlimited) by calling ulimit -c unlimited in the shell, which starts the process.
The cwd is writable by the segfaulting process.

Then you can analyze the file with

gdb <exe> <corefile>

answered Dec 20 '16 at 09:19

Ctx

18,090
24
36
51

To generate corefile, do I have to "build" the program with some debug informations? – Martin Perry Dec 20 '16 at 09:23
While it is helpful (compile with `-ggdb`), because the corefile contains more information then (for example to show the line number of the crashing source code line), it is not absolutely necessary. – Ctx Dec 20 '16 at 09:25
@MartinPerry NO, you have to increase only process resource limit. Do run only on console what Ctx told. "ulimit -c unlimited" – Sumit Gemini Dec 20 '16 at 09:25

score 3 · Accepted Answer · answered Dec 20 '16 at 09:30

Since your code not crashing every time, you can use backtrace as well. Using this you can see the function call stack at the time of crash. There are many examples available. In my projects I normally use the following code for backtracing.

/*
*   call reg_usr2 function from main
*   gcc -rdynamic myfile.c -o output
*/


#include <stdio.h>
#include <stdarg.h>
#include <signal.h>
#include <unistd.h>
#include <stdlib.h>
#include <execinfo.h>

#define FILE_NAME   "/tmp/debug"
#define MODE        0xFFFF

void dbgprint(int flag, char* fmt, ...)
{
    if(flag & MODE) {
        char buf[100];
        va_list vlist;
        FILE *fp = fopen(FILE_NAME,"a");
        va_start(vlist, fmt);
        vsnprintf( buf, sizeof( buf), fmt, vlist);
        va_end( vlist);
        fprintf(fp,"[%x]->%s\n", flag, buf);
        fclose(fp);
    }
}

/** Here is the code to print backtrace **/
void print_stack_trace ()
{
    void *array[20];
    size_t size;
    char **strings;
    size_t i;

    size = backtrace (array, 20);
    strings = backtrace_symbols (array, size);

    dbgprint(0xFFFF, "Obtained %zd stack frames.", size);
    dbgprint(0xFFFF, "-------------------------");
    dbgprint(0xFFFF, "---------Backtrace-------");

    for (i = 0; i < size; i++)
        dbgprint (0xFFFF, "%s", strings[i]);
    dbgprint(0xFFFF, "-------------------------");

    free (strings);
}

void sig_handler(int signo)
{
    FILE *fp = fopen(FILE_NAME,"a");
    if (signo == SIGUSR2){
        dbgprint(0xFFFF, "received SIGUSR2");
        dbgprint(0xFFFF, "----------------");
    }
    print_stack_trace();
    exit(0);
}

void reg_usr2()
{
    if (signal(SIGUSR2, sig_handler) == SIG_ERR)
        printf("\ncan't catch SIGUSR2\n");
}

int main()
{
    reg_usr2(); //should be first line of main after variables

    //Code.....

    return 0;  
}

Sorry, I am not aware of core dump. can you please give any link for it? — shafeeq, Dec 20 '16 at 09:41
Thanks for your great information. But I am working in a linux based machine which has no support of this. That is why I have only backtrace choice. — shafeeq, Dec 20 '16 at 09:58
NO it is only for linux machine. see second answer which Ctx gave. — Sumit Gemini, Dec 20 '16 at 10:01
execinfo may not available on all linux based system such as android, also the program should be linked with -rdynamic — e.jahandar, Dec 20 '16 at 10:49

score 3 · Answer 3 · edited May 23 '17 at 12:08

3

You can generate backtrace by catching SIGSEGV signal, and see where your application throw an invalid access. see https://stackoverflow.com/a/77336/4490542

But there is more easier solution, try running your application with catchsegv

catchsegv './program args'

and better alternative, valgrind

valgrind --tool=none ./program args

edited May 23 '17 at 12:08

Community

1
1

answered Dec 20 '16 at 09:54

e.jahandar

1,715
12
30

Linux - segmentation fault only sometimes - how to debug

3 Answers3