c# double to character array or alternative

Question

I have a program which stores a bunch of structs instances containing many members of type double. Every so often i dump these to a file, which I was doing using string builder e.g. :

 StringBuilder builder = new StringBuilder(256);

 builder.AppendFormat("{0};{1};{2};", x.A.ToString(), x.B.ToString(), x.C.ToString()); 

where 'x' is an instance of my type, and A,B,C are members of X of type double. I call ToString() on each of these to avoid boxing. However, these calls to ToString still allocate lots of memory in the context of my application, and i'd like to reduce this. What i'm thinking is to have a character array, and write each member direct into that and then create one string from that character array and dump that to the file. Few questions:

1) Does what i'm thinking of doing sound reasonable? Is there anything anyone knows of that would already achieve something similar?

2) Is there already something built in to convert double to character array (which i guess would be up to some paramaterised precision?). Ideally want to pass in my array and some index and have it start writing to there.

The reason i'm trying to do this is to reduce big spikes in memory when my app is running as I run many instances and often find myself limited by memory.

Cheers A

Answer 1

Is the file required to be some kind of text format?

If not, by far the most efficient way to do this is using a BinaryWriter (and BinaryReader to read them back).

See http://msdn.microsoft.com/en-us/library/system.io.binarywriter.aspx for more information.

Answer 2

If writing to the text file directly is possible, the steamwriter can be used to write strongly typed structs. I haven't tested memory usage, but I reckon they should be efficient

        using (var tw = new System.IO.StreamWriter("filename", true)) //(true to append to file)
        {
            tw.Write(x.A);
            tw.Write(';');
        }

If a stringbuilder is required, strongly typed overloads can also be called by using:

        builder.Append(x.A) //strongly typed as long as the field is a system type
            .Append(';')
            .Append(x.B)
            .Append(';'); 

Of course both methods would look better implementing some sort of loop or delegates, but that's beside the boxing logic.

EDIT custom double writing posted in other answer: c# double to character array or alternative

Answer 3

You should write directly to the file stream to reduce memory utilization.

using(var writer = new StreamWriter(...))
{
   writer.Write(x.A);
   writer.Write(";");
   writer.Write(x.B);
   writer.Write(";");
   writer.Write(x.C);
}

Answer 4

Are you sure that it is the presumable many calls to Double.ToString that is causing your memory problems? Each string should be collected on next generation 0 collection and the .NET garbage collector is pretty efficient in doing this.

If the strings you create exceed 85K they will be created on the large object heap and this may increase the total memory required by your application even though the large strings only exists transiently (large object heap fragmentation).

You can use Performance Monitor to learn more about how your application uses the managed heap. You have used CLRProfiler which is an even more advanced tool so maybe you wont learn anything new.

StringBuilder is the right class for building strings in memory but if you only build the strings in memory to later write it to a file you should instead write directly to the file using a StreamWriter.

StringBuilder will have to extend the buffer used to store the string and you can avoid this extra overhead by setting the capacity of the StringBuilder in advance (you already do this in your sample code).

No matter what overload you call to format a Double into a StringBuilder the call will eventually result in Double.ToString being called. StringBuilder.AppendFormat formats directly into the buffer without allocating an extra formatted string so in terms of memory usage StringBuilder.AppendFormat is just as fine as StringBuilder.Append and both overloads will allocate a string with the formatted Double as part of the formatting process. However, StringBuilder.AppendFormat will box the Double because it accepts an params Object[] array. Using the StringBuilder.Append overload that accepts a Double does not suffer from this problem.

If you with certainty knows that Double.ToString is the source of your memory problems I believe that you best option is to write your own floating point formatting code that can write a floating point number directly to a StringBuilder. The task is non-trivial but you could get inspiration from an open source C library.

Answer 5

Out of sheer curiosity on how to go about, I couldn't resist trying to create a scenario that would write doubles directly. Below is the result. I haven't benchmarked it or anything, but it did work as expected in the (limited) tests I've run.

        double[] test = { 8.99999999, -4, 34.567, -234.2354, 2.34, 500.8 };
        using (var sw = new FileStream(@"c:\temp\test.txt", FileMode.Create))
        {
            using (var bw = new BinaryWriter(sw))
            {
                const byte semicol = 59, minus = 45, dec = 46, b0 = 48;

                Action<double> write = d =>
                {
                    if (d == 0)
                        bw.Write(b0);
                    else
                    {
                        if (d < 0)
                        {
                            bw.Write(minus);
                            d = -d;
                        }

                        double m = Math.Pow(10d, Math.Truncate(Math.Log10(d)));
                        while(true)
                        {
                            var r = ((decimal)(d / m) % 10); //decimal because of floating point errors
                            if (r == 0) break;
                            if (m == 0.1)
                                bw.Write(dec); //decimal point
                            bw.Write((byte)(48 + r));         
                            m /= 10d;
                        }
                    }

                    bw.Write(semicol);
                };

                foreach (var d in test)
                    write(d);
            }
        }