Interesting OutOfMemoryException with StringBuilder

Question

I have the need to continuously build large strings in a loop and save them to database which currently occasionally yields an OutOfMemoryException.

What is basically going on here is I create a string using XmlWriter with StringBuilder based on some data. Then I call a method from an external library that converts this xml string to some other string. After that the converted string is saved to the database. This whole thing is done repeatedly in a loop about a 100 times for different data.

The strings by itself are not too big (below 500kByte each) and the process memory is not increasing during this loop. But still, occasionally I get a OutOfMemeoryExcpetion within StringBuilder.Append. Interestingly this exception does not result in a crash. I can catch that exception and continue the loop.

What is going on here? Why would I get an OutOfMemoryException although there is still enough free memory available in the system? Is this some GC heap problem?

Given that I can't circumvent converting all these strings, what could I do to make this work reliably? Should I force a GC collection? Should put a Thread.Sleep into the loop? Should I stop using StringBuilder? Should simply retry when confronted with a OutOfMemoryException?

score 17 · Accepted Answer · edited Aug 18 '20 at 12:50

17

There is memory but no contiguous segment that can handle the size of your string builder. You have to know that each time the buffer of the string builder is too short, its size is doubled. If you can define (in the ctor) the size of your builder, it's better. You MAY call GC.Collect() when you are done with a large collection of objects.

Actually, when you have an OutOfMemory, it generaly shows a bad design, you may use the hard drive (temp files) instead of memory, you shouldn't allocate memory again and again (try to reuse objects/buffers/...).

I STRONGLY advice you to read this post “Out Of Memory” Does Not Refer to Physical Memory from Eric Lippert.

edited Aug 18 '20 at 12:50

OfirD

9,442
5
47
90

answered Nov 20 '09 at 09:57

Guillaume

12,824
3
40
48

If I use the same StringBuilder over and over again, would I be using the same memory segement (except when the SB needs to enlarge itself)? Wouldn't that solve all my problems (most likely)? – bitbonk Nov 20 '09 at 10:11
I'm not sure, you will have to try to see if the string builder keeps it's capacity when cleared... – Guillaume Nov 20 '09 at 10:18
Not totaly pointless : you can avoid memory fragmentation which seems to be the true problem here. – Guillaume Nov 20 '09 at 13:25
Well it can re-allocate but if there is no other SB in garbage, it will be much more efficient. – Guillaume Nov 23 '09 at 08:40
A SB has a buffer, if you reuse the SB, you can reuse the buffer. If you creates a new SB each time, you create a new buffer each time and old buffers are in garbage. Even if the reused string builder is sized-up, there is a lot less garbage as there is only buffers from this builder and not all buffers from all string builders of previous operations. – Guillaume Nov 26 '09 at 09:14

score 3 · Answer 2 · answered Nov 20 '09 at 10:12

Try to reuse StringBuilder object when you do data generation.

After or before use just reset the size of the StringBuilder to 0 and start appending. This will decrease number of allocations and possibly make OutOfMemory situation very rare.

To illustrate my point:

void MainProgram()
{
    StringBuilder builder = new StringBuilder(2 * 1024); //2 Kb

    PerformOperation(builder);
    PerformOperation(builder);
    PerformOperation(builder);
    PerformOperation(builder);
}

void PerformOperation(StringBuilder builder)
{
    builder.Length = 0;

    //
    // do the work here builder.Append(...);
    //
}

score 3 · Answer 3 · answered Nov 20 '09 at 10:49

3

With the sizes you mention you are probably running into Large Object Heap (LOH) fragmentation.

Reusing StringBuilder objects is not a direct solution, you need to get a grip on the underlying buffers.
If possible, calculate or estimate the size beforehand and pre-allocate.

And it could help if you round up allocations, let's say to multiples of 20k or so. That could improve reuse.

answered Nov 20 '09 at 10:49

H H

263,252
30
330
514

Why is reusing StringBuilder objects not a direct solution? Do you just mean that reusing + inital preallocation (of a size that fits all future strings) would be the solution? – bitbonk Nov 20 '09 at 11:18
bitbonk, yes, something like that. The worst pattern for LOH is allocating blocks in increasing size and/or alternating short an long lived blocks. – H H Nov 20 '09 at 11:48

Interesting OutOfMemoryException with StringBuilder

3 Answers3

Linked