Which is the best way to improve memory usage when you collect a large data set before processing it? (.NET)

Question

When I have to get GBs of data, save it on a collection and process it, I have memory overflows. So instead of:

 public class Program
 {
     public IEnumerable<SomeClass> GetObjects()
     {
         var list = new List<SomeClass>();
         while( // get implementation
             list.Add(object);
         }
         return list;
     }

     public void ProcessObjects(IEnumerable<SomeClass> objects)
     {
         foreach(var object in objects)
             // process implementation
     }

     void Main()
     {
         var objects = GetObjects();
         ProcessObjects(objects);
     }
 }

I need to:

 public class Program
 {
     void ProcessObject(SomeClass object)
     {
         // process implementation
     }

     public void GetAndProcessObjects()
     {
         var list = new List<SomeClass>();
         while( // get implementation
             Process(object);
         }
         return list;
     }

     void Main()
     {
         var objects = GetAndProcessObjects();
     }
 }

There is a better way?

What kind of data is it? A database table, an ascii file, an xml file? — tuinstoel, Jun 23 '09 at 18:22
Do you have to be able to reason about the whole set at once? That is, can you get a small chunk, process a small chunk and then repeat instead of having the whole thing in memory at once? — JP Alioto, Jun 23 '09 at 18:23
Seconding JP's question, do you need the List< > in the second sample? — H H, Jun 23 '09 at 18:30
@Jader: That depends entirely on your "get implementation" that is not provided. You should change your "get implementation" to get the data in chunks, process the chunk, save the chunk and repeat if you can. — JP Alioto, Jun 23 '09 at 18:55

score 9 · Accepted Answer · answered Jun 23 '09 at 18:24

You ought to leverage C#'s iterator blocks and use the yield return statement to do something like this:

 public class Program
 {
     public IEnumerable<SomeClass> GetObjects()
     {
         while( // get implementation
             yield return object;
         }
     }

     public void ProcessObjects(IEnumerable<SomeClass> objects)
     {
         foreach(var object in objects)
             // process implementation
     }

     void Main()
     {
         var objects = GetObjects();
         ProcessObjects(objects);
     }
 }

This would allow you to stream each object and not keep the entire sequence in memory - you would only need to keep one object in memory at a time.

Didn't solve my problem. When I do Table.InsertAllOnSubmit(objects), the memory overflow still occurs. — Jader Dias, Jun 23 '09 at 18:44
http://stackoverflow.com/questions/1034429/how-to-prevent-memory-overflow-when-using-an-ienumerablet — Jader Dias, Jun 23 '09 at 18:51

score 6 · Answer 2 · answered Jun 23 '09 at 18:23

6

Don't use a List, which requires all the data to be present in memory at once. Use IEnumerable<T> and produce the data on demand, or better, use IQueryable<T> and have the entire execution of the query deferred until the data are required.

Alternatively, don't keep the data in memory at all, but rather save the data to a database for processing. When processing is complete, then query the database for the results.

answered Jun 23 '09 at 18:23

John Saunders

160,644
26
247
397

Now I will have to ask how to implement your IQueryable solution. – Jader Dias Jun 23 '09 at 18:39
Using any of the LINQ providers will return an IQueryable. – John Saunders Jun 23 '09 at 22:21

score 3 · Answer 3 · answered Jun 23 '09 at 18:29

3

public IEnumerable<SomeClass> GetObjects()
     {

       foreach( var obj in GetIQueryableObjects
             yield return obj
     }

answered Jun 23 '09 at 18:29

Rony

9,331
2
22
22

I think you meant "yield return". – Andrew Hare Jun 23 '09 at 18:29

score 1 · Answer 4 · answered Jun 23 '09 at 18:25

1

You want to yield!

Delay processing of your enumeration. Build a method that returns an IEnumerable but only returns one record at a time using the yield statement.

answered Jun 23 '09 at 18:25

Randolpho

55,384
17
145
179

Ralph Wiggum · Answer 5 · 2009-06-23T19:09:43.393

The best methodology in this case would be to Get and Process in chunks. You will have to find out how big a chunk to Get and Process by trial and error. So the code would be something like :

public class Program

{ public IEnumerable GetObjects(int anchor, int chunkSize) { var list = new List(); while( // get implementation for given anchor and chunkSize list.Add(object); } return list; }

 public void ProcessObjects(IEnumerable<SomeClass> objects)
 {
     foreach(var object in objects)
         // process implementation
 }

 void Main()
 {
     int chunkSize = 5000;
     int totalSize = //Get Total Number of rows;
     int anchor = //Get first row to process as anchor;
     While (anchor < totalSize)
     (
         var objects = GetObjects(anchor, chunkSize);
         ProcessObjects(objects);
         anchor += chunkSize;
     }
 }

}

Which is the best way to improve memory usage when you collect a large data set before processing it? (.NET)

5 Answers5