Segmentation error with caffe

Question

I am using caffe on Windows, and am getting segmentation errors which I cannot pinpoint. It happens when the program exits, and WinDbg said scalar deleting destructor, no idea where the memory was allocated. My complete code (currently a dummy code trying to narrow it down, but it happens only sometimes):

#include <string>
#include <vector>

#include "boost/algorithm/string.hpp"
#include "google/protobuf/text_format.h"
#include <stdio.h>

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include "caffe/blob.hpp"
#include "caffe/common.hpp"
#include "caffe/net.hpp"
#include "caffe/proto/caffe.pb.h"
#include "caffe/util/db.hpp"
#include "caffe/util/format.hpp"
#include "caffe/util/io.hpp"

using caffe::Blob;
using caffe::Caffe;
using caffe::Datum;
using caffe::Net;
using std::string;
namespace db = caffe::db;

int main(int argc, char** argv) {
    // Initialize logging with program call. First thing that needs to be done
    ::google::InitGoogleLogging(argv[0]);

    cv::Mat mean_;

    // Set Caffe to run in CPU only mode
    Caffe::set_mode(caffe::Caffe::CPU);

    std::vector<std::string> labels_;

    /*std::shared_ptr<Net<float>> caffe_test_net;*/
    Net<float>* caffe_test_net;
    caffe_test_net = new Net<float>("D:\\Development\\caffe-windows\\models\\bvlc_reference_caffenet\\deploy.prototxt", caffe::Phase::TEST);
    caffe_test_net->CopyTrainedLayersFrom("D:\\Development\\caffe-windows\\models\\bvlc_reference_caffenet\\bvlc_reference_caffenet.caffemodel");

    delete caffe_test_net;
    return 1;
}

I have tested with caffe_net in a unique or shared_ptr, but that made no difference at all. I am at a loss on how to find the issue at hand.

Answer 1

"Happens sometimes" is a pretty common thing with undefined behavior, which is what you're really encountering. A segmentation fault is one of a theoretically infinite number of things that the computer might do - the behavior is literally undefined. In other words, as they say on USENET: "It is legal for the compiler to make demons fly out of your nose." It may work, it might do something strange, or it might throw some major error like a segfault.

There are tools dedicated specifically to tracking down segmentation faults and other memory errors. On Linux, that's generally Valgrind, while on Windows, you'd use Dr. Memory. So long as you compiled with the debugging symbols included (-g), when you run the executable through Dr. Memory, it should give you a stack trace for the segmentation fault.

As soon as you get the stack trace, check the top of it to see which destructor the code is whining about, or in the very least, what line of code in main.cpp is calling the function(s) responsible for the undefined behavior.

Also, depending on your compiler, you may be encountering a known bug in VC.

You can find more general information about segmentation faults, common causes, and how to debug them on this answer.