List.clear Memory Problems

Question

I have a program that processes a bunch of csv files with many columns so that the user can pull out specific columns and write them to another csv file that's much smaller, and only contains what they require. Since there can be up to 8 million rows, I've created a queue system using a list, where it adds pieces(50000 lines at a time) to the list buffer so that the writer can write that piece, and then use listbuffer[i].clear(); to remove it, so that i am using much less memory, and also limit the buffer to only contain 10 items, keeping total memory use as low as i require it.

My problem is that thelistbuffer[i].clear(); is not letting go of the memory it previously used until the entire program is finished and I've reinitialized the buffer so that it is blank for the next use.

**listbuffer is aList<List<List<string>>>

Just so this makes a little more sense...

listbuffer[chunk of lines][row #][column #]

Is there a way to tell the list to let go of the memory that list[i] is using?

Thanks!

I think you need to post more about the problem than that, because `List.Clear` does indeed clear out any references it has to the objects in the list. — Lasse V. Karlsen, Jun 13 '12 at 15:05
See [related SO post on forcing GC](http://stackoverflow.com/a/233647/175679). — SliverNinja - MSFT, Jun 13 '12 at 15:07
Please post the problem code. At first glance `listbuffer[i].clear();` seems strange considering normally you would use `listbuffer.Clear()` to clear the entire array before filling it again. Why don't you just simply wait until the buffer is empty to clear it's memory? The big problem is that your examples appear to have syntax problems — Security Hound, Jun 13 '12 at 15:09
It sounds like each line is processed independently; if this is the case, then why all the manual buffering? — Eric Smith, Jun 13 '12 at 15:11
Calling `.Clear()` does not mean immediate release of related memory. The `GC` will collect it when it feels like it. — Magnus, Jun 13 '12 at 15:14
Try GC.Collect(), but you have to make sure that the objects that you are clearing from the buffer are not referenced any where else. statements like Object o = new Object; list.Add(o); list.Clear(); will not release the memory held by o unless you do o = null or dispose. — ata, Jun 13 '12 at 15:16

score 6 · Accepted Answer · edited Oct 10 '17 at 15:00

6

Yes, using List<T>.Clear() clears the references to all objects in the list and sets the Count property to 0. It does not release the reference to the underlying array that stores the object references. You can set the Capacity property to 0 after clearing it to release the reference to that array.

If you actually have OOM problems then doing this is not exactly a cure, it can create more address space fragmentation trouble. Check this answer for hints on how to avoid creating too much garbage in the Large Object Heap.

edited Oct 10 '17 at 15:00

Ehsan Sajjad

61,834
16
105
160

answered Jun 13 '12 at 15:24

Hans Passant

922,412
146
1,693
2,536

Thanks, That explained a lot. I assumed that the `GC.Collect()` was called each clear, so i was wondering why it was having problems. – Axxelsian Jun 13 '12 at 15:28
I decided to change the program a bit and use `list.remove(0)`, so my writer is always just using the first item in the buffer, then limit its size so i'n not using as much memory. Thanks for the extensive answer. It was Extremely helpful. – Axxelsian Jun 13 '12 at 18:13

score 3 · Answer 2 · edited Oct 10 '17 at 15:01

3

maybe it's too late, but I wrote this extension to clear memory from large lists.

It worked for me and I hope it helps.

Extension:

public static void LiberarLista<T>(this List<T> lista)
{
     lista.Clear();            
     int identificador = GC.GetGeneration(lista);
     GC.Collect(identificador, GCCollectionMode.Forced);
}

Example:

List<YourObject> hugeList= LetsRockBigData();

// Your code logic here 
// Finish... Let's clear the huge list 

hugeList.LiberarLista();

I apologize for not being expert in English. Only one DotNerd from Brazil and that's all folks...

edited Oct 10 '17 at 15:01

Ehsan Sajjad

61,834
16
105
160

answered Dec 16 '16 at 21:45

Vitor - DotNerd

41
4

This works nicely, but it is sloooowwww. when using a this in a loop that constantly adds large bodies of data, I noticed a slow down by about 400%. – Wilson212 Jun 13 '17 at 21:47
Hi, @Wilson212 thanks for your feed back! I used it in Parallel.ForEach and did not identify too much delay. I'll check and see if I can add more performance to this code. – Vitor - DotNerd Jun 26 '17 at 20:43
This will force all the objects in the generation of the heap to be released, not only your list. That is the slowness of clearing it., it's too much processor intensive – jefissu Apr 14 '23 at 17:31

Oswaldo Junior · Answer 3 · 2019-10-25T14:32:43.557

-2

You must clear the list, and after that invoke the GC to remove it from memory. As follows with extensive method:

public static void ClearMemory<T>(this List<T> lista)
    {
        int identificador = GC.GetGeneration(lista);
        lista.Clear();
        GC.Collect(identificador, GCCollectionMode.Forced);
    }

edited Oct 25 '19 at 14:32

answered Oct 25 '19 at 14:27

Oswaldo Junior

51
3

List.clear Memory Problems

3 Answers3