sb = new StringBuilder(102410241024) > 'System.OutOfMemoryException' was thrown

Question

I need to append lots of string in a loop and this will create a very long string.
1GB should be long enough to store the string, therefore I do this:

var sb = new StringBuilder(1024 * 1024 * 1024);

but receives this error

'System.OutOfMemoryException' was thrown

any advice to append long strings?

I'm writing a backup tools for MySQL database. The tools will export data from database and write into a text file. The data contains BLOB and TEXT data type. So, will expect very long string.

This is the code: Code Updated 1

var conn = new MySqlConnection(Program.ConnectionString);
var cmd = new MySqlCommand();
cmd.Connection = conn;
conn.Open();

cmd.CommandText = selectSQL;

MySqlDataReader rdr = cmd.ExecuteReader();

while (rdr.Read())
{
    var sb = new StringBuilder(1024 * 1024 * 1024);
    for (int i = 0; i < rdr.FieldCount; i++)
    {
        if (sb.Length > 0)
            sb.AppendFormat(separator);

        if (rdr[i].GetType() == typeof(byte[]))
        {
            sb.AppendFormat(ConvertByteArrayToHexString((byte[])rdr[i]));
        }
        else
            sb.AppendFormat(rdr[i] + "");
    }
    WriteToFile(sb.ToString());
}

conn.Close();

It appears that your JVM is not configured to have access to 2 gigabytes of memory (1G of chars == 2GB). — Sergey Kalinichenko, Dec 11 '13 at 01:18

score 6 · Answer 1 · answered Dec 11 '13 at 01:18

6

Use a file stream to write long files. There's no need to keep the full string in memory at any one time.

answered Dec 11 '13 at 01:18

recursive

83,943
34
151
241

score 4 · Accepted Answer · edited May 23 '17 at 10:24

The out of memory error may be caused by any number of reasons, based on how Windows virtual memory and how the .NET runtime works.

Do you really need to dump everything into the StringBuilder all at once? If you're writing to a file, you can do the operation in pieces at a time.

Use something like this as a starting point: int bufferSize = 65536;

using (StreamWriter sw = new StreamWriter("filename", true, System.Text.Encoding.UTF8, bufferSize))
{
    while(!finished)
    {
        string data = "foo"; //get next data here...
        sw.Write(data);
    }
}

`

References: What consumes less resources and is faster File.AppendText or File.WriteAllText storing first in StringBuilder?

Updated the code. Amazing how the one-liners still get the most votes. — marknuzz, Dec 11 '13 at 01:34

score 4 · Answer 3 · edited May 23 '17 at 12:18

You haven't asked for a StringBuilder with enough capacity to store 1 GB - you've asked for one with enough capacity to store 1024 * 1024 * 1024 characters, which'll require (roughly) 2 GB of memory. That's smaller than the maximum string length (2^31-1 characters), but larger than the heap space available in a 32-bit process.

You may not need a StringBuilder with that kind of capacity - others answers discuss that. Assuming you need a string that gigantic, you definitely don't need to pre-allocate your StringBuilder with that size. Just create a StringBuilder, and let it grow on its own:

var sb = new StringBuilder();

or pre-allocate something more reasonable, if you know you'll have huge amounts of data:

var sb = new StringBuilder(1024 * 1024);

sb = new StringBuilder(1024*1024*1024) > 'System.OutOfMemoryException' was thrown

3 Answers3

sb = new StringBuilder(102410241024) > 'System.OutOfMemoryException' was thrown