How to free memory which is allocated by a List in C#

Question

I have a code like this :

var myList = db.Table1.ToList();
/*doing some operations on the list*/
var myList = db.Table2.ToList();
/*again doing some operations on the list*/
var myList = db.Table3.ToList(); // I'm getting out of memory exception here.

I can't retrieve data by pages because I need all of the table once. How can I dispose( I mean free the space which is alloacted by that list) list before I load another table? Thanks.

EDIT :

I'm actually generating many (sometimes thousands of) sublists from myList after I load it. So I really need to learn how to free a list.

EDIT 2 : Here is my full Stacktrace :

   at System.Collections.Generic.List`1.set_Capacity(Int32 value)
   at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
   at System.Collections.Generic.List`1.Add(T item)
   at System.Data.Entity.Core.Objects.EntityEntry.TakeSnapshot(Boolean onlySnapshotComplexProperties)
   at System.Data.Entity.Core.Objects.Internal.SnapshotChangeTrackingStrategy.TakeSnapshot(EntityEntry entry)
   at System.Data.Entity.Core.Objects.Internal.EntityWrapper`1.TakeSnapshot(EntityEntry entry)
   at System.Data.Entity.Core.Objects.ObjectStateManager.AddEntry(IEntityWrapper wrappedObject, EntityKey passedKey, EntitySet entitySet, String argumentName, Boolean isAdded)
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper.HandleEntityAppendOnly[TEntity](Func`2 constructEntityDelegate, EntityKey entityKey, EntitySet entitySet)
   at System.Data.Entity.Core.Common.Internal.Materialization.Coordinator`1.ReadNextElement(Shaper shaper)
   at System.Data.Entity.Core.Common.Internal.Materialization.Shaper`1.SimpleEnumerator.MoveNext()
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at System.Collections.Generic.List`1.AddRange(IEnumerable`1 collection)
   at WebApplication2.MyMethod in line 2292
   at WebApplication2.Controllers.MyController.MyActtion(String myString) in line 137
   at System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters)
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f()

Possible duplicate of [How to Dispose a list of objects;Does it Free the Memory](https://stackoverflow.com/questions/32412641/how-to-dispose-a-list-of-objectsdoes-it-free-the-memory) — Samvel Petrosov, May 25 '17 at 12:34
_I need all of the table once. How can I dispose list before I load another table?_ two sentences are complete opposite. why do you want to dispose list? — M.kazem Akhgary, May 25 '17 at 12:34
Side note: why don't perform your operation at database side without retrieving whole (possible huge) table to client? — Andrey Korneyev, May 25 '17 at 12:35
What happens if you do `Table3` first? Does it fit in memory? — Sergey Kalinichenko, May 25 '17 at 12:37
@AndyKorneyev Because I'm doing very complex operations on db on C# side. It would be too hard to do them on db side using SQL. — jason, May 25 '17 at 12:38
@dasblinkenlight Yes it does, but it's not the point, I need to work on whole table once so I need to free some memory before I call next table. — jason, May 25 '17 at 12:40
So how much rows are in those tables? Is that application runs as 32-bit or 64-bit? — Evk, May 25 '17 at 12:41
@jason Try setting `myList` to `null` before calling `db.Table3.ToList()` — Sergey Kalinichenko, May 25 '17 at 12:44
Using `.ToList` leaves the possibility that the created list might be up to twice as large as it needs to be (doubles in size every time the capacity limit is reached). If you know the size up front, `new` up a list of the right capacity and copy into it. The sizing algo looks like : `int newCapacity = _items.Length == 0? _defaultCapacity : _items.Length * 2;` It also means that for ~1000000 items, it will need to be grown about 18 times (from a default initial size of 4). This will cause a lot of memory thrashing (and therefore, fragmentation). — spender, May 25 '17 at 12:44
Well 1m is not much, should be fine even with 10m. However, there is not much you can do. Setting to null won't help, unless you first set to null and then force garbage collection before each `myList` initialization (with `GC.Collect`). Ensure also that you run in release mode without debugger attached. — Evk, May 25 '17 at 12:48
@spender while that is true - OP claims to run in 64-bit, so fragmentation should not be a problem. — Evk, May 25 '17 at 12:51
may be of help: https://stackoverflow.com/questions/1852929/can-i-force-memory-cleanup-in-c — , May 25 '17 at 12:57
@Evk When I run in release mode, I'm getting internal server error. — jason, May 25 '17 at 13:03
Well with that we cannot help, just look up in the logs of your server what goes wrong, that is not related to this question. — Evk, May 25 '17 at 13:05
@spender can you come up with an answer of your comment please? I want to do your way, since there seems no other option. — jason, May 25 '17 at 13:24
@jason Did it work out for you? If yes, I'll post as the answer. — spender, May 25 '17 at 14:03
I couldn't try, could you post it as the answer with some more detail please? — jason, May 25 '17 at 14:04
Posting the full exception call stack trace might help identifying the actual problem. — Ivan Stoev, May 27 '17 at 13:50
@IvanStoev it's just an Out Of Memory exception of .NET and it's because those lists take too much space. I need to find a way to free them. — jason, May 27 '17 at 14:18
Well, that's your guess. `OutOfMemory` exception does not always mean you have no available memory, and all your lists (if not used/stored somewhere) will be garbage collected. The problem could even be in EF (or whatever the `db.Table` is). Exception *stack trace* gives additional information for the actual operation causing the exception, and is a good starting point. — Ivan Stoev, May 27 '17 at 15:56
I think you need to clarify your question. For instance, if `db` variable is a EF `DbContext` shared between the 3 calls, even if you clear, set to `null` and force GC on your lists, the `DbContext` will still hold all these objects in it's local cache lists, thus killing all your efforts to "free" your lists. But that's just a guess. And w/o additional information all you'll get will be a guesswork. — Ivan Stoev, May 27 '17 at 16:07
Put list work (allocation) in a separate function. Once you're done with a list and out of that function, try a GC.Collect. If it doesn't change then you need to understand what objects are kept in memory beetwen two lists computation. (https://blogs.msdn.microsoft.com/johan/2007/01/11/i-am-getting-outofmemoryexceptions-how-can-i-troubleshoot-this/). If you want to keep all these lists and items in memory, then buy more RAM. — Simon Mourier, May 27 '17 at 16:12
@IvanStoev That is true, db is shared between 3 calls. I guess that's why setting to null and calling GC doesn't work. I'm posting complete stack trace. Thank you. — jason, May 27 '17 at 16:27
Good. So the problem is definitely inside `DbContext` tracker data structures. You could try `new`-ing the `db` variable before populating the next list. Or use `AsNoTracking` option, i.e. `var myList = db.Table1.AsNoTracking().ToList();` (same for other `ToList` calls). — Ivan Stoev, May 27 '17 at 16:37
@IvanStoev it would be great if you give it as an example with some explanation :) then you would get the bounty. Just using `.AsNoTracking()` to all `ToList()`s would work? — jason, May 27 '17 at 16:43
@IvanStoev AsNoTracking worked like charm. Thank you so much! — jason, May 27 '17 at 16:48
@Jason You are welcome, glad that we were able to identify and resolve the issue, good cooperative job :) — Ivan Stoev, May 27 '17 at 17:16
Can you explain "/*again doing some operations on the list*/" part so we can understand what are you trying to do?. — Miguel, May 28 '17 at 16:40

Ivan Stoev · Accepted Answer · 2017-05-27T17:09:22.977

Based on the additional clarifications in the post comments, the db is EF DbContext derived class variable, shared between the 3 calls. And Table1, Table2 and Table3 are DbSet instances inside that context.

What happens when you use var myList = db.Table1.ToList() is that additionally to reading the table in memory and filling up your list, the EF DbContext also populates the internal DbSet.Local collection with the same objects (so called tracking) in order to be able to detect and apply the changes you made (eventually) to the underlying objects.

And here is the problem. Event if you Clear, set to null the myList variable, the internal cache (list) still holds all these objects, eventually causing the OutOfMemoryException at some point.

So the trick is to either totally eliminate the DbContext internal caching by using so called No-Tracking Queries if you are not planning to modify the returned objects and save them back to the database:

var myList = db.Table1.AsNoTracking().ToList();

or clean up (Dispose and set to null) and use fresh new context (db = new YourDbContext()) for each list process.

Of course you could combine the above with the other techniques mentioned in the comments for cleaning up your own list variables.

This is a rather tricky point and your answer helped me a lot. Thank you! — Alexander Dayan, Dec 28 '22 at 21:37

score 0 · Answer 2 · answered May 25 '17 at 14:08

0

As per my comment above, ToList can be quite thrashy on memory when dealing with large number of items.

As such, you can sometimes get better memory performance by pre-allocating the List.

So:

var tableCount = db.Table1.Count();
var myList = new List<Table1Item>(tableCount); //I don't know the type parameter here
myList.AddRange(db.Table1); //or db.Table1.AsEnumerable() ?

answered May 25 '17 at 14:08

spender

117,338
33
229
351

Thanks but I still get the exception. – jason May 25 '17 at 14:25

How to free memory which is allocated by a List in C#

2 Answers2