0

I have to parallelize an application; This application has to process a file, I used tasks and each thread is processing one line of data (previously it used to process lines one by one sequentially). The problem is that the data provider uses some sort of caching and also it accesses the file multiple times during the process, so using one data provider was a hard work of making it thread-safe, instead I created on new data provider for each time a thread is processing a line. It works fine for the first couple of runs, but after some time the COM object which is used to access the file starts throwing the following error:

COMException occurred: ErrorCode = -2147287036

As I checked "2147287036" is the error code for "not enough resources to open another file". I'm sure I'm calling the close/dispose for the data providers/COM objects, so I have a hard time understanding why I can not access the file (I have an exception handling logic that tries to open the file in the write mode and if it is unsuccessful, it tries to open it in the read mode, and again if it is unsuccessful it throws the exception which is basically the one mentioned above).

My first clue is that the COM object doesn't release the file handle instantly. But still that doesn't make that much sense.

DeveloperInToronto
  • 1,173
  • 2
  • 14
  • 32
  • 1
    why do you say that your assumption (that the handle is not being released) doesn't make sense? Seems to me a pretty good explanation, you can test if resources are being leaked by inspecting the handles count in the task manager - if this number is continously increasing it means that the file handles are not being released – floyd73 Mar 09 '11 at 14:08
  • Because I explicitly close it; but I'll check the handles count and see how it is changing. Thanks. – DeveloperInToronto Mar 09 '11 at 14:13
  • 1
    you might run into the limit of the number of open files if all of these data provides don't close their handles immediately – BrokenGlass Mar 09 '11 at 14:15
  • @floyd @BrokenGlass actually you guys are right. The handles just increase. The COM object is of type IStorage and the way it is being released is by calling Marshal.ReleaseComObject(...), but it seems it is not really releasing it. – DeveloperInToronto Mar 09 '11 at 16:27
  • 1
    sorry I never used a COM object directly in managed code but have you tried FinalReleaseComObject? From MSDN: "If you want to call this method to ensure that a COM component is released at a determined time, consider using the FinalReleaseComObject method instead." – floyd73 Mar 09 '11 at 19:58

2 Answers2

1

Wouldn't it be better to have a producer/consumer setup where a single producer reads in the file line by line and feeds that into a queue where the consumer threads can go all out?

Opening the same file from multiple threads to do heavy crunching just sounds a bit cumbersome.

Sorry if I miss-understood your issue.

Bengie
  • 1,035
  • 5
  • 10
  • Yes, I agree. If I was going to write the application from scratch I would definitely go with that solution; but this is a big application handed to me to be multi-threaded, and it is lots of code. So I'm trying to test the quick solutions first. – DeveloperInToronto Mar 09 '11 at 16:20
0

I found the issue. There was a memory leak, in some parts of the code some copies of the COM object was being created and not released and apparently Marshal.ReleaseComObject(...) decrements the counter corresponding to the number of references to the COM object and only releases it if that number becomes 0.

DeveloperInToronto
  • 1,173
  • 2
  • 14
  • 32