Properway to dispose of class in Parallel.Foreach

Question

I have a simple Parallel.Foreach loop that has about 1000 rows in the DataTable, each of these Rows calls a new Class, however, the memory builds up until i run out of memory. I'm wondering how to you dispose to a new Class properly in regards to a parallel. If you're saying what a newb question it's because Parallel and Threading is new to me.

  var options = new ParallelOptions();
        options.MaxDegreeOfParallelism = 5;
        Parallel.ForEach(urlTable.AsEnumerable(),options, drow =>
        {
            WebSiteCrawlerClass WCC = new WebSiteCrawlerClass();
            if (drow.ItemArray[0].ToString().Contains("$"))
            {

                WCC.linkGrabberwDates(drow.ItemArray[0].ToString(), "www");
            }
            else
            {
                WCC.NoDatesCarCrawler(drow.ItemArray[0].ToString(), "www");
            }
        });

score 0 · Accepted Answer · answered Apr 05 '12 at 22:52

0

If WebSiteCrawlerClass is disposable then you would do this

    using( var WCC = new WebSiteCrawlerClass() )
    {
        if (drow.ItemArray[0].ToString().Contains("$"))
        {
            WCC.linkGrabberwDates(drow.ItemArray[0].ToString(), "www");
        }
        else
        {
            WCC.NoDatesCarCrawler(drow.ItemArray[0].ToString(), "www");
        }
    }

answered Apr 05 '12 at 22:52

Phil

42,255
9
100
100

okay, let me make the new dumb question, how do I know or how do i make a class disposable – Apr 05 '12 at 22:54
It doesn't need to be disposable unless it holds references to instances of other disposable types, or non-managed resources. The methods of the class need to dispose any disposable object they create as well. Search for IDisposable/Disposable pattern. – Phil Apr 05 '12 at 22:56
so basically WebSiteCrawlerClass() : IDisposable – Apr 05 '12 at 22:57
Yes but just adding a Dispose method won't help unless you use is correctly. – Phil Apr 05 '12 at 22:59

score 0 · Answer 2 · answered Apr 05 '12 at 23:07

0

This sort of problem can be better addressed with async instead of Parallel. Dish out all the requests, and process them as they come back to ya. Just a thought.

answered Apr 05 '12 at 23:07

GregC

7,737
2
53
67

I've done similar things in F#, but it translates to C#/Async pretty well. http://stackoverflow.com/questions/4431743/map-reduce-with-f-agents/4438256#4438256 – GregC Apr 05 '12 at 23:08

score 0 · Answer 3 · answered Apr 05 '12 at 23:45

What you want to do is have a single WebSiteCrawlerClass object for each thread instead of initializing one for each loop. This can be done using the 'localInit' overload of Parallel.ForEach. Something like:

    var options = new ParallelOptions();
    options.MaxDegreeOfParallelism = 5;
    Parallel.ForEach(urlTable.AsEnumerable(),
      options,
      () => new WebSiteCrawlerClass(),
      (drow, dummyLoopState, WCC) =>
      {
        if (drow.ItemArray[0].ToString().Contains("$"))
        {
            WCC.linkGrabberwDates(drow.ItemArray[0].ToString(), "www");
        }
        else
        {
            WCC.NoDatesCarCrawler(drow.ItemArray[0].ToString(), "www");
        }
        return WCC;
      },
      (wcc) => { } );

This assumes your WebSiteCrawlerClass object is reusable. If you need to reset its state or something, that is done during the Finally delegate at the end (so something like (wcc) => { wcc.Reset(); } at the end).

Properway to dispose of class in Parallel.Foreach

3 Answers3