3

I am using Visual C# Express 2008 and I have an application that starts up on a form, but uses a thread with a delegated display function to take care of essentially all the processing. That way my form doesn't lock up while tasks are being processed.

Semi-recently, after going through a repeated process a number of times (the program processes incoming data, so when data comes in, the process repeats) my app will crash with a System.OutOfMemory error.

The stack trace in the error message is useless because it only directs me to the the line where I call the delegated form control function.

I've heard people say they use ProcMon from SysInternals to see why errors like this happen. But I, for the life of me, can't figure it out. The amount of memory I am using doesn't change as the program runs, if it goes up, it comes back down. Plus, even if it was going up, how do I figure out which part of my program is the problem?

How can I go about investigating this problem?

EDIT:

So, after delving further into this issue, I looked through anything that I was ever re-declaring. There were a few instances where I had hugematrix = new uint[gigantic], so I got rid of about 3 of those.

Instead of getting rid of the error, it is now far more obscured and confusing.

My application takes the incoming data, and renders it using OpenGL. Now, instead of throwing "System.OutOfMemory" it simply does not render anything with OpenGL.

The only difference in my code is that I do not make new matrices for holding the data I plot. That way, I hope, my array stays in the same place in memory and doesn't do anything suicidal to my LOH.

Unfortunately, this twists the beast far beyond my meager means. With zero errors popping up, and all my data structures apparently still properly filled, how can I find my problem? Does OpenGL use memory in an obscure way so as to not throw exceptions when it fails? Is memory still a problem? How do I find out? All the memory profilers in the world seem to tell me very little.

EDIT:

With the boatloads of support from this community (with extra kudos to Amissico) the error has finally been rooted out. Apparently I was adding items to an OpenGL list, and never taking them off the list.

The app that finally clued me in was .Net Memory Profiler. At the time of crash it showed 1.5GB of data in the <unknown> category. Through process of elimination (everything else in the list that was named), the last thing to be checked off the list was the OpenGL rendering pipleline. The rest is history.

Gorchestopher H
  • 1,141
  • 3
  • 17
  • 35
  • Does it always crash on the same line? – Mongus Pong Mar 05 '10 at 14:23
  • 1
    It would be nice to see some code. Where is the code to show what the delegate is actually doing? – Ryan Alford Mar 05 '10 at 14:25
  • Can you provide more information about how the incoming data is retrieved ? Synchronously/Asynchronously ? Event ? Polling ? Can you also tell how do your graphic and worker thread communicate ? – Seb Mar 05 '10 at 14:28
  • Some sample code would be helpful – Jason Mar 05 '10 at 14:29
  • It *claims* to always crash on the same line, but it's just on the delegated function call. On the Invoke. I could show you code, but it's pretty beastly... There's hundreds of lines, and functions are being called all over the place. Essentially the Delegate function unpacks an item from a structure list, takes an image out, displays it to the screen. Then it fills some data into various textboxes, updates a ZedGraph Control, and passes a bitmap to an OpenGL object. I assume the problem could be anywhere in there. – Gorchestopher H Mar 05 '10 at 14:33
  • @Seb The data comes in via some proprietary function by MVTec. Essentially they interface with an industrial depth gauge array. The data coming in is a 1024x720 array of long ints. The data is retrieved synchronously, and happens via an event. I am not sure how to properly answer the communication question, but I am using a delegate fucntion, and an invoke command. – Gorchestopher H Mar 05 '10 at 14:38
  • Based on the size of the array, I strongly suspect Large Object Heap Fragmentation as Jacob G mentioned. – Paul Williams Mar 05 '10 at 14:52
  • If I do garbage cleanup with System.GC.Collect() should that fix up the LOH? I put that line in my code to run after every new set of data is loaded. I still get the memory issue. – Gorchestopher H Mar 06 '10 at 20:12
  • System.GC.Collect() really doesn't help in all the cases I have run against. Even though it doesn't help, I always live the statement in with a comment why I tried to collect. – AMissico Mar 06 '10 at 20:19
  • I agree. Array size is a suspect. Can you intialize once, then use many times by re-initializing the values and not the whole array? – AMissico Mar 06 '10 at 20:21
  • ProcMon is not going to help at all. For this stuff your need a profiler such as "CLR Profiler". – AMissico Mar 06 '10 at 20:22
  • @AMissico I will try initializing my arrays once, instead of re-initializing them every time their "size" needs to change. – Gorchestopher H Mar 06 '10 at 21:28
  • 1
    Maybe create one array as large as the maximum you expect. Then maintain varibles that hold the current valid array bounds. – AMissico Mar 06 '10 at 22:56
  • @AMissico, I followed your suggestion and have done this with my large arrays. I still seem to be missing something. I've edited my original post with more info. – Gorchestopher H Mar 08 '10 at 13:59

5 Answers5

5

Based on the description in your comments, I would suspect that you are either not disposing of your images correctly or that you have severe Large Object Heap fragmentation and, when trying to allocate for a new image, don't have enough contiguous space available. See this question for more info - Large Object Heap Fragmentation

Community
  • 1
  • 1
Jacob G
  • 3,645
  • 1
  • 20
  • 25
  • I have a feeling you are exactly correct... I don't have too many threads out there, and I do dispose of my images as soon as I don't need them. It's basically got to be this heap thing... That link has a lot of stuff I don't understand at all... but I guess it's time to start learning. I suppose I could just... not deallocate the memory I use, and keep putting it into the same place, so I don't get the fragmentation issue. – Gorchestopher H Mar 05 '10 at 15:02
  • You can read some more information here - http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/ ... One strategy is to serialize and deserialize the objects on the LOH. That will compact the heap, but it's likely a decent performance hit. – Jacob G Mar 05 '10 at 15:36
  • Can't you use just one image, and clear it then use it again for the next image? Why create and dispose? I assume each image is the same size. – AMissico Mar 06 '10 at 20:15
  • I notice you have a question about loading bitmaps into a picture box. Is the question in regards to this application? – AMissico Mar 06 '10 at 20:31
  • @AMissico Yes, this is the same application. My application is a little hairy... but essentially it is the same thing. I load a "version" of a bitmap into a picturebox using the solution to that question, but I also send that bitmap to some functions that generate massive point and color lists to render in OpenGL. So, I can't dispose of that bitmap for the life of me anywhere in my app. If I try, the picturebox will explode immediately. – Gorchestopher H Mar 06 '10 at 21:10
  • 1
    Yes, you must keep the picture box image. That is normal behavior. Need to see your bitmap creation. Can you post somewhere for review? If not post your code as answer for http://stackoverflow.com/questions/2189384/how-can-i-load-a-generated-bitmap-into-a-picturebox. – AMissico Mar 06 '10 at 23:00
  • Could I send you source files instead? – Gorchestopher H Mar 08 '10 at 02:04
  • This looks like the most appropriate answer. Now I just need to bash my head around the fixes. Thanks for your help everyone. – Gorchestopher H Mar 08 '10 at 16:55
1

You need to use a memory profiler, such as the ants memory profiler to find out what causes this error.

Are you re-registering an event handler on every loop and not un-registering it?

Oded
  • 489,969
  • 99
  • 883
  • 1,009
  • I am not really running on a loop. I guess I need to get my terminology straight. Basically it's just an event handler, just one. So, when I said loop up there, I meant that the event is getting triggered multiple times. – Gorchestopher H Mar 05 '10 at 14:29
  • I have just started using RedAnts, and I can not see any dead giveaways. – Gorchestopher H Mar 06 '10 at 21:25
1

CLR Profiler for the .NET Framework 2.0 at https://github.com/MicrosoftArchive/clrprofiler

The most common cause of memory fragmentation is excessive string creation.

Glorfindel
  • 21,988
  • 13
  • 81
  • 109
AMissico
  • 21,470
  • 7
  • 78
  • 106
0

Following considerations:

  1. Make sure that threads you spawn are destroyed (aborted or function return). Too much threads can fail application, although in Task Manager used memory is not too high
  2. Memory leaks. Yes, yes, you can cause them in .net pretty well without setting reference to nulls. This can be solved by using memory profilers like dotTrace or ANTS Memory Profiler
Andrey
  • 59,039
  • 12
  • 119
  • 163
  • dotTrace seems to close along with my app, when it crashes. I'm most likely using it incorrectly. – Gorchestopher H Mar 05 '10 at 15:15
  • I've done some more work with dotTrace, and I've taken a snapshot of my app after 1 iteration of incoming data, and 10 iterations. This has told me almost nothing, as the memory snapshots seem to indicate that I'm not doing anything wrong. – Gorchestopher H Mar 06 '10 at 19:26
  • Can you post the "snapshot" data for us to review? – AMissico Mar 06 '10 at 20:26
  • I am not familar with dotTrace but it must have summary reports. – AMissico Mar 06 '10 at 20:27
  • I would love to post any kind of snapshot that would help people help me. ...but I don't know how. I have snapshots using dotTrace, RedAnts, and most recently .Net Memory Profiler. As a preview, .net Memory Profiler complains that I have a mouse event handler... but I only create it once. – Gorchestopher H Mar 06 '10 at 21:12
0

I had an OutOfMemoryException-problem as well: Microsoft Visual C# 2008 Reducing number of loaded dlls

The reason was fragmentation of 2GB GB virtual address space and poster nobugz suggested Sysinternal's Vmmap utility which has been very helpful for diagnostics. You can use it to check if your free memory areas become more fragmented over time. (First sort by size then by type -> refresh repeat sorting and you can see if contiguous free memory blocks become smaller)

Community
  • 1
  • 1
user282727
  • 674
  • 1
  • 8
  • 21
  • By using the utility I can see nothing obvious. Although, I am not seeing where there contiguous free blocks are... Am I allowed to post my memory VMMap snapshots on here? Could someone take a look and let me know what they're seeing? – Gorchestopher H Mar 07 '10 at 17:19
  • My "largest free" statistic starts off around 270M, goes to around 128M after 10 cycles, then back up to 270M after 20 cycles. I'm really not seeing it... I've heard this exception can happen if you're mismatching types as well. – Gorchestopher H Mar 07 '10 at 17:22
  • Interesting, if it goes up again it sounds like fragmentation is not the problem. So I would suggest putting fragmentation at the end of the list for possible problem causes and concentrate on other possibilities. – user282727 Mar 07 '10 at 20:27
  • When profiling with .Net Memory Profiler it claims that my mouse event handler is a possible issue. I am not sure why this would be, as there is only ever a single instance (which is never closed) of the object that event handles. If this problem has no good way to track down, is there some kind of aggressive memory mashing I can do in order to force my app to reset it's memory back to start conditions? That way I can do that after every 10 iterations or so? – Gorchestopher H Mar 08 '10 at 02:03