Allocate large static std::unordered_map results in stack overflow

Question

I am creating a static std::unordered_map as follows:

 auto* __epsgMap__ = new std::unordered_map <int/*EPSG*/, CRS::Info> ({

{3819, CRS::Info("HD1909","+proj=longlat +ellps=bessel +towgs84=595.48,121.69,515.35,4.115,-2.9383,0.853,-3.408 +no_defs")},
{3821, CRS::Info("TWD67","+proj=longlat +ellps=aust_SA +no_defs")},
{3824, CRS::Info("TWD97","+proj=longlat +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +no_defs")},
{3889, CRS::Info("IGRS","+proj=longlat +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +no_defs")},
{3906, CRS::Info("MGI 1901","+proj=longlat +ellps=bessel +towgs84=682,-203,480,0,0,0,0 +no_defs")},
{4001, CRS::Info("Unknown datum based upon the Airy 1830 ellipsoid","+proj=longlat +ellps=airy +no_defs")},
...
...
});

This map has 5k+ entries, hence, on MSVC, allocating it results in stack overflow (I guess it pushes an std::initializer_list on the stack).

How can I allocate this, witout successive calls to insert()?

** Edit **

Tried to do this in an allocater function with successive calls to std::unordered_map::insert() but is overflows the stack anyway when the allocator function is called (it probably pushes all the elements onto the stack anyway)

I'm not clear on what's causing the problem. Is the process of simply pushing the arguments to the stack causing it to overflow? — François Andrieux, Jul 11 '17 at 13:23
The content of this map looks like it can be stored in sorted array with const initialization. — user7860670, Jul 11 '17 at 13:25
@FrançoisAndrieux I guess. This does not happen under GCC/LLVM. — manatttta, Jul 11 '17 at 13:25
Is it in function scope? That huge initializer results in a huge std::initializer_list. If that in turn has automatic storage duration, you'll have problems. — StoryTeller - Unslander Monica, Jul 11 '17 at 13:26
@StoryTeller yes, the initializer list goes onto the stack, that's what's crashing. Can I initialize this in any other way without pushing items on the stack? — manatttta, Jul 11 '17 at 13:27
Is the data sorted? Does the map receive new entries at run-time? — StoryTeller - Unslander Monica, Jul 11 '17 at 13:30
@FrançoisAndrieux that's not relevant to the case, I guess. Is a simple struct of 2 strings. — manatttta, Jul 11 '17 at 13:31
Store it in a `const static` array and fill up the map with repeated `insert`, possibly calling `reserve` beforehand; it's not like the current method fares any better in terms of static memory consumption or runtime performance. Or, as suggested by others, if the data is static just put it into a static sorted array and do your lookups using binary search. — Matteo Italia, Jul 11 '17 at 13:32
@StoryTeller what do you mean by sorted? It's in an unordered_map and I want to keep it that way for lookup performance reasons — manatttta, Jul 11 '17 at 13:32
Visual Studio traditionally allocates a pretty small stack (~1M) but you can make it larger if you need to. https://stackoverflow.com/questions/20744650/how-to-overcome-stack-size-issue-with-visual-studio-running-c-codes-with-big-ar — Retired Ninja, Jul 11 '17 at 13:33
@RetiredNinja that sounds like a bad option since this is not stored on the stack (by design of std::unorderd_map). It's only the initializer list that temporarily is pushed onto the stack — manatttta, Jul 11 '17 at 13:34
@manatttta I did not say it was a good option or the best option, only an option. — Retired Ninja, Jul 11 '17 at 13:35
If you target a 64 bit system, 1m stacks are pretty silly small. Stack isn't allocated, its address range is just reserved, and 64 bit systems have a lot of address space to play with. 1/2^44 is tiny; even a 1 gigabyte stack would only take 1/2^34 of the process's address space. — Yakk - Adam Nevraumont, Jul 11 '17 at 15:53

StoryTeller - Unslander Monica · Accepted Answer · 2017-07-11T14:22:26.773

Seems the problem is only with regards to where the initializer for the map resides. I suggest you move it to a static array, and use the appropriate unordered_map constructor to create the map:

static std::pair<int/*EPSG*/, CRS::Info> const map_init[] = {

{3819, CRS::Info("HD1909","+proj=longlat +ellps=bessel +towgs84=595.48,121.69,515.35,4.115,-2.9383,0.853,-3.408 +no_defs")},
//...
};

auto* epsgMap = new std::unordered_map <int/*EPSG*/, CRS::Info>(std::begin(map_init), std::end(map_init));

BTW, identifiers beginning with a pair of leading underscores are reserved for the implementation, always. I therefore removed them my code sample.

Also, I don't know why you need to dynamically allocate the map, but C++ function local static objects are always initialized only when control first passes through them. Since std::unordered_map already allocated memory by itself, the extra allocation seems redundant. You may want to change you map to a value instead:

static std::unordered_map<int/*EPSG*/, CRS::Info> epsgMap(
   std::begin(map_init), std::end(map_init)
);

Allocate large static std::unordered_map results in stack overflow

1 Answers1