Error message while in C++ program

Question

I am storing around 14GB of data in following map:

struct data
{
   char* a;
   char* b;
   data(char* _a, char* _b)
   {
        int alen = strlen(a);
        a = new char[alen+1];
        strcpy(a,_a);
        a[alen]='\0';

        int blen = strlen(_b);
        b = new char[ blen+1];
        strcpy(b,_b);
        b[blen]='\0';        
    }

    ~data()
    {
       delete [] a;
       delete [] b;
    }
};

struct ltstr
{
    bool operator()(const char* s1, const char* s2) const
    {
        return strcmp(s1, s2) < 0;
    }
};

map<const char*, data*, ltstr> m;

Program runs for certain number of records(10440440 out of 26293289) and after some time i a m getting following error message.

Program terminated with signal SIGKILL, Killed.
The program no longer exists. 
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6.x86_64 libgcc- 4.4.5-6.el6.x86_64 libstdc++-4.4.5-6.el6.x86_64

how can i avoid above termination?

Server Specification:

  Mem:  24605344k total, 15148556k used,  9456788k free,    20892k buffers
  Swap:  2097144k total,   161364k used,  1935780k free, 14469244k cached

Do you have 14 gigs of RAM or even 14 gigs of RAM plus swap? — Mark, Jan 11 '13 at 02:13
Failure to comply with the rule of three could result in segfaults and possible expulsion from the planet. — chris, Jan 11 '13 at 02:15

score 4 · Answer 1 · edited May 23 '17 at 11:51

4

The OOM killer is probably stepping in and killing your process. Take a look in your logs to be sure.

edited May 23 '17 at 11:51

Community

1
1

answered Jan 11 '13 at 02:11

Oliver Charlesworth

267,707
33
569
680

Jan 11 09:31:00 kernel: Killed process 3016 () vsz:25668380kB, anon-rss:23738008kB, file-rss:564kB – Manish Jan 11 '13 at 02:22
1

I just added server specification also. – Manish Jan 11 '13 at 03:24

score 1 · Answer 2 · answered Jan 11 '13 at 02:14

1

Without knowing more details of your program you might try memory mapping the file to reduce memory usage.

answered Jan 11 '13 at 02:14

Brian Ensink

11,092
3
50
63

score 1 · Answer 3 · answered Jan 11 '13 at 02:22

Since you're apparently running out of memory, the obvious thing to do is to reduce the amount of memory you're using. One possibility would be to use something like a Patricia trie to store the data instead of a std::map (which normally uses a red-black tree).

Exactly how much that'll gain will depend on just how much redundancy you have in your strings. To an extent, it'll also depend on the size of the strings you're using as keys, versus the size of the data you're storing with them.

Depending on the situation, you might also consider using something like Huffman compression with a fixed table for the strings you're storing as data associated with each key. Although Huffman compression isn't necessarily the most effective in other situations, with a fixed table it has the advantage of working reasonably well with individual strings like you're working with here. If the strings in the data part are long (e.g., average of at least 8K for the two strings) it might well be worth applying an LZ* family compression in addition to the Huffman compression, but if they're short (e.g., anything less than at least a few kilobytes) LZ probably won't work very well (unless you're willing to group the strings together into blocks, so you might have to decompress a few kilobytes of other strings to get to the one you care about at any given time when you're retrieving some data.

While compression will normally be slower than accessing the data directly from memory without compressing/decompressing, it will still usually be faster than retrieving data from disk, as will typically happen if you run out of physical memory and end up using virtual memory.

score 1 · Accepted Answer · answered Jan 11 '13 at 02:31

1

how can i avoid above termination?

By rethinking your design. What are you trying to accomplish? In particular, why do you need a lookup table with 27 million entries in memory?

answered Jan 11 '13 at 02:31

arayq2

2,502
17
21

Error message while in C++ program

4 Answers4