Difficult to track SIGSEGV Segmentation fault in large program

Question

I apologise for posting a question that has been asked many times (I've just read 10 pages of them) but I can't find a solution.

I'm working on a multi-threaded graphic/audio program using OpenGL and Portaudio respectively. The audio thread uses a library I'm making for audio processing objects. The SIGSEGV happens maybe 20% of the time (much less when debugging) and happens when resetting loads of audio objects with new stream information (sample rate, vector size etc). Code::blocks Debugger states the fault as originating from different places each time the fault happens.

This is the audio processing loop:

while(true){
    stream->tick();
    menuAudio.tick();
    {
        boost::mutex::scoped_lock lock(*mutex);
        if(channel->AuSwitch.resetAudio){
            uStreamInfo newStream(channel->AuSwitch.newSrate, 
                 channel->AuSwitch.newVSize, channel->AuSwitch.newChans);
            menuAudio.resetStream(&newStream);
            (*stream) = newStream;
            menuAudio.resetStream(stream);
            channel->AuSwitch.resetAudio = false;
        }
    }
}

It checks information from the graphics thread telling it to reset the audio and runs the resetStream function of the patch object, which is basically a vector for audio objects and runs each of them:

void uPatch::resetStream(uStreamInfo* newStream)
{
    for(unsigned i = 0; i < numObjects; ++i){
/*This is where it reports this error: Program received signal SIGSEGV,  
Segmentation fault. Variables: i = 38, numObjects = 43 */
        objects[i]->resetStream(newStream); 
    }
}

Sometimes it states the SIGSEGV as originating from different locations, but due to the rarity of it faulting when run with the debugger this is the only one I could get to happen.

As there are so many objects, I won't post all of their reset code, but as an example:

void uSamplerBuffer::resetStream(uStreamInfo* newStream)
{
    audio.set(newStream, false);
    control.set(newStream, true);
    stream = newStream;
    incr = (double)buffer->sampleRate / (double)stream->sampleRate;
    index = 0;
}

Where the audio.set code is:

void uVector::set(uStreamInfo* newStream, bool controlVector)
{
    if(vector != NULL){
        for(unsigned i = 0; i < stream->channels; ++i)
            delete[] vector[i];
        delete vector;
    }
    if(controlVector)
        channels = 1;
    else
        channels = newStream->channels;
    vector = new float*[channels];
    for(unsigned i = 0; i < channels; ++i)
        vector[i] = new float[newStream->vectorSize];
    stream = newStream;
    this->flush();
}

My best guess would be that it's a stack overflow issue, as it only really happens with a large number of objects, and they each run fine individually. That said, the audio stream itself runs fine and is run in a similar way. Also the loop of objects[i]->resetStream(newStream); should pop the stack after each member function, so I can't see why it would SIGSEGV.

Any observations/recommendations?

EDIT:

It was an incorrectly deleted memory issue. Application Verifier made it fault at the point of the error instead of the occasional faults identified as stemming from other locations. The problem was in the uVector stream setting function, as the intention of the class is for audio vectors using multidimensional arrays using stream->channels, with the option of using single dimensional arrays for control signals. When deleting to reallocate the memory I accidentally set all uVectors regardless of type to delete using stream-> channels.

if(vector != NULL){
    for(unsigned i = 0; i < stream->channels; ++i)
        delete[] vector[i];
    delete vector;
}

Where it should have been:

if(vector != NULL){
    for(unsigned i = 0; i < this->channels; ++i)
        delete[] vector[i];
    delete vector;
}

So it was deleting memory it shouldn't have access to, which corrupted the heap. I'm amazed the segfault didn't happen more regularly though, as that seems like a serious issue.

score 0 · Accepted Answer · edited May 23 '17 at 10:24

0

I you can spare the memory, you can try a tool like Electric Fence (or DUMA, its child) to see if it's an out of bound write that you perform. Usually these types of segfaults (non-permanent, only occurring sometimes) are relics of a previous buffer overflow somewhere.

You could try Valgrind also, which will have the same effect as the 2 tools above, to the cost of a slower execution.

Also, try to check what's the value of the bad address you're accessing when this happens: is it looking valid? Sometimes a value can be very informative on the bug you're encountering (typically: trying to access memory at 0x12 where 0X12 is the counter in a loop :)).

For stack overflows... I'd suggest trying to increase the stack size of the incriminated thread, see if the bug is reproduced. If not after a good bunch of tries, you've found the problem.

As for windows:

edited May 23 '17 at 10:24

Community

1
1

answered Apr 26 '12 at 06:32

Gui13

12,993
17
57
104

Very informative, thank you. This is primarily a Windows project at the moment, and I've seen Valgrind recommended a lot for this sort of issue. Do you know of any freely available alternatives for Windows? I tried increasing the stack size to no real effect (it took ages to reproduce the fault, but that's usually the case). – Utopia Apr 26 '12 at 07:11
Ok so that's most probably that your pointer is either not valid anymore (deleted by another thread?) or that you wrote something out of bounds beforehands and it only shows up there. As for tools, SO has some answers that I'll list in my post :) – Gui13 Apr 26 '12 at 09:34
Thanks for the links. Got it working! I tried application verifier and instead of faulting at various points some times it faulted on the first iteration of the resetStream every time, so tracking down the problem was much easier. I'll update my post with the problem/solution. :) – Utopia Apr 27 '12 at 00:08

score 0 · Answer 2 · answered Apr 26 '12 at 06:41

I think you just made it a Stack Overflow issue. :)

In all seriousness, bugs like these are usually the result of accessing objects at memory locations where they no longer exist. In your first code block, I see you creating newStream on the stack, with a scope limited to the if statement it is a part of. You then copy it to a dereferenced pointer (*stream). Is safe and correct assignment defined for the uStreamInfo class? If not explicitly defined, the compiler will quietly provide memberwise copy for object assignment, which is OK for simple primitives like int and double, but not necessarily for dynamically allocated objects. *stream might be left with a pointer to memory allocated by newStream, but has since been deallocated when newStream went out of scope. Now the data at that RAM is still there, and for a moment will look correct, but being deallocated memory, it could get corrupted at any time, like just before a crash. :)

I recommend paying close attention to when objects are allocated and deallocated, and which objects own which other ones. You can also take a divide an conquer approach, commenting out most of the code and gradually enabling more until you see crashes starting to occur again. The bug is likely in the most recently re-enabled code.

Thanks for the response. In the uStreamInfo struct there isn't any dynamically allocated memory, so I assumed a memberwise copy would be okay. Admittedly the multiple resetStream() using &newStream and then back to stream is bad code, but it should still be valid and run. — Utopia, Apr 26 '12 at 06:54

Difficult to track SIGSEGV Segmentation fault in large program

2 Answers2