Segment fault exception when try to add large number of elements to a static map

Question

Problem: Segment fault exception when try to add large number of elements to a map.

Below program just runs fine, BUT if I add around 80000 elements to it, it just crashes at run time. From what I googled around, this is due to limited stack size available to a program. Any workaround ? Basically I just need to keep a map of cityId to City name of ~90000 cities .

Utilities.hpp

#ifndef Utilities_hpp
#define Utilities_hpp
#include <string>
#include <map>

class Utilities {

public:
    typedef std::map<long, const char *> CityMap;
    static CityMap cityIdNameMap;

public:
    static std::string getCurrencySymbol(std::string &$currenyName);
    static const char *getCityFromId(long cityId);
    static CityMap createMap();
};
#endif /* Utilities_hpp */

Utilities.cpp

#include "Utilities.hpp"

Utilities::CityMap Utilities::cityIdNameMap = Utilities::createMap();

Utilities::CityMap Utilities::createMap() {
    CityMap tmpMap = {
        {1000010,"Abu Dhabi,Abu Dhabi,United Arab Emirates"},
        {1000010,"Abu Dhabi,Abu Dhabi,United Arab Emirates"},
        {1000011,"Ajman,Ajman,United Arab Emirates"}
    };

    return tmpMap;
}

// http://stackoverflow.com/questions/13464325/static-map-initialization
//class Utilities {
std::string Utilities::getCurrencySymbol(std::string &$currenyName) {
    return "$";
}

const char *Utilities::getCityFromId(long cityId) {
    return Utilities::cityIdNameMap[cityId];
}

main.cpp

#include <iostream>
#include <istream>
#include "Utilities.hpp"

using std::cout;
using std::endl;


int main(int argc, char *argv[])
{
    cout << Utilities::getCityFromId(1000011);

    return 0;
}

Segment fault log:

System Integrity Protection: disabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_PROTECTION_FAILURE at 0x00007fff57bf8ff8

VM Regions Near 0x7fff57bf8ff8:
MALLOC_SMALL           00007fbf5c000000-00007fbf5c800000 [ 8192K] rw-/rwx SM=PRV  
--> STACK GUARD            00007fff543f9000-00007fff57bf9000 [ 56.0M] ---/rwx SM=NUL  stack guard for thread 0
Stack                  00007fff57bf9000-00007fff583f9000 [ 8192K]   rw-/rwx SM=COW  thread 0

Application Specific Information:
/*********/CSVParser/DerivedData/Build/Products/Debug/./CSVParser

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   CSVParser                       0x00000001079e53a3 Utilities::createMap() + 1955795 (Utilities.cpp:13)

1   CSVParser                       0x0000000108da2384 __cxx_global_var_init + 20 (Utilities.cpp:11)

2   CSVParser                       0x0000000108da23b9 _GLOBAL__sub_I_Utilities.cpp + 9

3   dyld                            0x00007fff6bd6410b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265

4   dyld                            0x00007fff6bd64284 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40

5   dyld                            0x00007fff6bd608bd ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305

6   dyld                            0x00007fff6bd60743 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 127

7   dyld                            0x00007fff6bd609b3 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75

8   dyld                            0x00007fff6bd530f1 dyld::initializeMainExecutable() + 208

9   dyld                            0x00007fff6bd56d98 dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 3596

10  dyld                            0x00007fff6bd52276 dyldbootstrap::start(macho_header const*, int, char const**, long, macho_header const*, unsigned long*) + 512

11  dyld                            0x00007fff6bd52036 _dyld_start + 54

Can you show us the broken code? Working code doesn't really help us spot the problem. — NathanOliver, Sep 26 '16 at 11:40
segment fault log added, as I said same program crashes when I add ~ 80k elements in createMap function. And of course all elements has different key. — Piyuesh, Sep 26 '16 at 11:50
So you had all 80000 strings embedded as literals in your code? Put them into a file and read it in instead. — , Sep 26 '16 at 11:55
Ok, but I need to be able to fetch city name of a city ID efficiently, how will it happen when I read them from file. Am I missing something obvious ? — Piyuesh, Sep 26 '16 at 12:01
@Piyuesh: If you read your elements from a file into a map, accessing them through the map will still be efficient. Access time doesn't depend on how they first got into the map. — Josh Sanford, Sep 26 '16 at 12:07
@Piyuesh You read the file once at start up and if the number of entries becomes very large you use a dedicated database. Data does not belong in code. — , Sep 26 '16 at 12:07
Ok thanks, I'll try this approach (read from file) and update you with results. Just curious: will stack reach limit won't happen if we read entries from file ? — Piyuesh, Sep 26 '16 at 12:17

score 2 · Accepted Answer · answered Sep 26 '16 at 12:06

The map is allocated in the heap, but what I guess is happening is the initializer list is being allocated on the stack, hence the segfault.

Alternatively, you could:

Read the ID city pairs from a file instead of hardcoding it.
If you require very fast performance for createMap, you could try boost::serialization approach to serialize the map to binary file and deserialize it - make sure to measure if it is really faster.

Segment fault exception when try to add large number of elements to a static map

1 Answers1