Rapidjson seg fault with g++ optimization

Question

I am using Rapidjson and have noticed that when I turn on optimization in g++ (-O1/-O2/-O3) I am getting a segmentation fault. I think I have tracked it down to the GenericValue& AddMember() function within rapidjson.

GenericValue& AddMember(GenericValue& name, GenericValue& value, Allocator& allocator) {
    RAPIDJSON_ASSERT(IsObject());
    RAPIDJSON_ASSERT(name.IsString());

    Object& o = data_.o;
    if (o.size >= o.capacity) {
        if (o.capacity == 0) {
            o.capacity = kDefaultObjectCapacity;
            o.members = reinterpret_cast<Member*>(allocator.Malloc(o.capacity * sizeof(Member)));
        }
        else {
            SizeType oldCapacity = o.capacity;
            o.capacity += (oldCapacity + 1) / 2; // grow by factor 1.5
            o.members = reinterpret_cast<Member*>(allocator.Realloc(o.members, oldCapacity * sizeof(Member), o.capacity * sizeof(Member)));
        }
    }
    o.members[o.size].name.RawAssign(name);
    o.members[o.size].value.RawAssign(value);
    o.size++;
    return *this;
}

When debugging, I can see that kDefaultObjectCapacity ( is being optimized out (this is a static const SizeType kDefaultObjectCapacity = 16)

Therefore the line "o.capacity = kDefaultObjectCapacity;" is not being executed, and the malloc is mallocing 0 bytes then trying to cast it.

Why is this static const being removed?

I have tried making Object& o both volatile and static, neither worked. Any ideas?

Thanks Will

EDIT: I can't easily run the tests as it's on an embedded platform, rapidjson is built using buildroot at the moment. I tried the unit tests but couldn't get them going on the target.

I can have a look at providing the assembly, but it's part of a big application so it might be difficult finding the right bit.

For info, this is the method that calls the rapidjson code and this is where the problem seems to be:

int16_t FrontEndJSONHandlers::get_run_cfg_packer(JSONEngine& json_engine, char *message, int32_t *length)
{
Document doc;

// Need to pack an empty request to get the data
doc.SetObject();
doc.AddMember(JSONRPC_MEMBER, JSONRPC_VERSION, doc.GetAllocator());
doc.AddMember(METHOD_MEMBER, JSON_RPC_METH_GET_RUN_CFG, doc.GetAllocator());
doc.AddMember(ID_MEMBER, json_engine.GetNextMessageID(), doc.GetAllocator());

// Format the message
json_engine.FormatMessageAndRegisterResponseHandler(&doc, &message, &length, get_run_cfg_handler);

return 0;
}

If I make Document doc static, it doesn't seg fault - not sure whether this is the best way around it though?

Answer 1

Your code looks pretty badly wrong, and in a way that makes me suspect you are also misunderstanding what you see in the debugger. Try fixing the bugs first, before worrying about what the optimizer might be doing:

It appears that part of your intent is for o.capacity to end up > (or maybe you intended >=) o.size

But did you consider the case o.size > kDefaultObjectCapacity when o.capacity starts at zero? More seriously, your comment about increasing by a factor of 1.5 is wrong for code that decreases capacity. You probably intended oldCapacity*3/2

if (o.size >= o.capacity) {
    if (o.capacity == 0) {
        o.capacity = kDefaultObjectCapacity;
        o.members = reinterpret_cast<Member*>(allocator.Malloc(o.capacity * sizeof(Member)));
    }
    else {
        SizeType oldCapacity = o.capacity;
        o.capacity += (oldCapacity + 1) / 2; // grow by factor 1.5
        o.members = reinterpret_cast<Member*>(allocator.Realloc(o.members, oldCapacity * sizeof(Member), o.capacity * sizeof(Member)));
    }
}

Edit: I was mistaken about a large part of the above. But I still think a misunderstanding of what was seen in the debugger is a more likely explanation that the optimizer breaking that code.

Was the "90" mentioned decimal, or might it have been hex? The optimizer certainly could skip multiplying by sizeof(Member) and go directly to the result of that multiplication.

Answer 2

I got a similar issue, compiling with GCC 4.9: I get a segmentation fault when an optimization flag is specified (-On). It's very easy to duplicate, here's the code:

//...
#include "extern/rapidjson/document.h"
//...
void* MyThread(void*)
{
   std::cout << "Start of MyThread" << std::endl;

   rapidjson::Document doc;
   doc.Parse("{ \"NAME\": \"Peter\", \"AGE\": 38, \"Male\": true}");

   std::cout << "End!";

   return NULL;
}

int main(int argc, char** argv) 
{
   pthread_t thread;

   pthread_create(&thread, NULL, &MyThread, NULL);

   std::string str;
   std::cout << "Press <Return>"  << std::endl;
   std::getline(std::cin, str);

   return 0;
 }

The key here is the document being created and parsed in a separate thread. If we call "MyThread(NULL)" directly in the main(), we won't get any bug.

A similar code with Visual Studio 2012, optimized, runs fine. I ran a VS code analysis and no error was found. The only difference is the threading:

std::thread th([] { MyThread(NULL); }); 
th.join();

===============================================

The issue has been fixed after version 1.0.2. Using the latest code works fine.