0

I've notice in my test code that using the ConcurrentQueue<> somehow does not release resources after Dequeueing and eventually I run out of memory. Or the Garbage collection is not happening frequently enough.

Here is a snippet of the code. I know the ConcurrentQueue<> store references and yes, I do want create a new object each time so if the enqueueing is faster than dequeueing, memory will continue to rise. Also a screenshot of the memory usage. For testing, I sent through 5000 byte arrays with 500000 elements each.

There is a similar question asked:

ConcurrentQueue holds object's reference or value? "out of memory" exception

and everything mentioned in that post is what I experienced ... except that the memory won't release after dequeueing, even when the Queue is emptied.

I would appreciate any thoughts/insights to this.

ConcurrentQueue<byte[]> TestQueue = new ConcurrentQueue<byte[]>();

Task EnqTask = Task.Factory.StartNew(() =>
{                
    for (int i = 0; i < ObjCount; i++)
    {
        byte[] InData = new byte[ObjSize];
        InData[0] = (byte)i; //used to show different array object
        TestQueue.Enqueue(InData);                                        
        System.Threading.Thread.Sleep(20);                    
    }
});

Task DeqTask = Task.Factory.StartNew(() =>
    {
        int Count = 0;
        while (Count < ObjCount)
        {
            byte[] OutData;
            if (TestQueue.TryDequeue(out OutData))
            {
                OutData[1] = 0xFF;  //just do something with the data
                Count++;                        
            }
            System.Threading.Thread.Sleep(40);
        }

Picture of memory

Community
  • 1
  • 1
pz1145
  • 29
  • 4
  • I put in the delay deliberately since in the real application, enqueuing will be faster than dequeuing but there will be a long time delay after a certain number of objects that allows the queue to clear up and the memory freed. The whole testing actually took ~40ms x 5000 = 3.33 minutes. The memory was never freed even after the tasks were finished. – pz1145 Jan 30 '17 at 00:09
  • You may want to read this.... http://stackoverflow.com/questions/10016541/garbage-collection-not-happening-even-when-needed I think basically the garbage collector doesn't collect until it actually needs to. As discussed here.... http://stackoverflow.com/questions/233596/best-practice-for-forcing-garbage-collection-in-c-sharp Collecting memory is an expensive operation and can cause performance issues. If your machine has lots of memory available, what's the issue with using it? – Mick Jan 30 '17 at 00:09
  • Is there any reason that you cannot use a BlockCollection as the first comment suggests in the link you posted in your question? – ninja coder Jan 30 '17 at 00:16
  • @Mick - The real application involves an industrial camera (100Hz) that sends image data (~500K) to the application for processing. Since the computer can't process each image that fast (10ms), it needs a buffer. If the queue cannot free resources by dequeuing, it will run out of memory no matter the available memory. – pz1145 Jan 30 '17 at 00:44
  • @ninjacoder - I will try using the BlockCollection as suggested. – pz1145 Jan 30 '17 at 00:49

0 Answers0