0

I am working on a project where in I have created a thread pool of 8 threads and each keep thread has its own queue where in data comes and then its picked by the individual threads and then processes it. Now, here as the thread stack size is 8MB, hence will my thread at any point run out of memory as these are never ending threads and these get killed only when the application goes down.

In these threads I have local STL containers( unnordered_map ) where in I keep on adding the data and also keep on deleting them. So, is there any chance by which my thread can report OOM error because its never ending one?...

Also, is there a way in which in c++11 we can increase the stack size of the thread?...

HolyBlackCat
  • 78,603
  • 9
  • 131
  • 207
Shailendra Patil
  • 11
  • 4
  • If you are using `std::thread` then you cannot change the size of the stack (see [this answer](https://stackoverflow.com/a/13871812/1982207)). – Ian Gralinski Jul 25 '20 at 14:24

1 Answers1

0

The fact that a thread's execution is never-ending does not mean that it will run out of stack. You see, the thread does not execute tasks by going deeper and deeper in the call stack. A worker thread essentially does this:

while (have_tasks_left()) {
    auto task = get_next_task();
    execute_a_task(task);
}

At any given time, the thread's stack only has a single task object.

Now, if you're worried about overflowing the thread's queue for tasks - that's not a stack problem actually. Your threads' queues have their basic data structure in thread-local storage, or perhaps even on the stack - but those queues allocate larger buffers off the stack (e.g. std::queue or std::priority_queue). Meaning that you are only limited by the amount of available system memory.

einpoklum
  • 118,144
  • 57
  • 340
  • 684
  • But, here each thread has its own queue, and into this queue I pump in packets from my main thread, and once the thread receives packet it pops it out from the queue and then processes the packet further, once it ends up processing, it again returns back and wait for next packet in the queue and the process continues. During this process I see that my application memory keeps on increasing and after sometime it's out of memory. It has used all the system memory. I also ran valgrind, but I could see 'possible lost' errors in Oracle libs which is not correct. So where is whole systm memory used? – Shailendra Patil Jul 25 '20 at 19:07
  • I am using ZMQ to receive the packets and then based on some hash calculation I put it in a specific threads queue....I have used one mutex and one condition variable to achieve this.... Even then I see my system memory of 16GB getting utilized fully and left with no memory...So I am not getting where is the memory leak happening... – Shailendra Patil Jul 25 '20 at 19:12
  • 1
    @ShailendraPatil: It's certainly possible that the work queues eventually exhaust system memory. But - it's not through thread stacks, it's through heap allocations, which - by default - are common to all threads. There's no contradiction between the allocations being used exclusively by a single threads each and them coming out of a single process-wide heap. – einpoklum Jul 25 '20 at 19:59