What is the efficient way to write stream of bytes into a file in c#?

Question

I have big big data in form of bytes around 5GB.

I need to store this data in a file ServerData.xml. This data should be first converted into string and then should be saved to file so that we can perform operation on the file.

I used below code to convert stream of bytes to string and then to save the same in a file.

private const string fileName = "ServerData.xml";

public void ProcessBuffer(byte[] receiveBuffer, int bytes)
{
    if (!File.Exists(fileName))
    {
        using (File.Create(fileName)) { };
    }

    TextWriter tw = new StreamWriter(fileName, true);
    tw.Write(Encoding.UTF8.GetString(receiveBuffer).TrimEnd((Char)0));
    tw.Close();
}

Is it the right way ?

or please suggest better way so that there should not be any memory issue if any in future ?

Answer 1

The code in your question can only work if ProcessBuffer is always called with a UTF-8 encoded text that is broken on code point boundaries. That seems pretty unlikely to me, so I would expect that you encounter errors when decoding to text.

However, decoding to text and then writing, is rather pointless and indeed counter-productive. The bytes are already UTF-8 encoded. Write them directly to file as they arrive from the socket. Don't perform any processing of them. When you come to read the XML using XmlReader, the parser will read the encoding as UTF-8 from the document's XML declaration, and be able to decode the rest of the document. I am assuming that the document's XML declaration specifies UTF-8 but that seems highly likely. You should check.

You should get rid of the text writer which is no use to you for writing bytes. Write the bytes directly to a file stream. And try to avoid opening and closing the file repeatedly. That's very inefficient. Open and close the file exactly once.

Answer 2

You can simply write these bytes to a file using FileStream:

public void ProcessBuffer(byte[] receivedBuffer, int bytes)
{
    using (var fileStream = new FileStream(fileName, FileMode.Create)) // overwrites file
    {
        fileStream.Write(receivedBuffer, 0, bytes);
    }
}

Update: You won't be able to work with such a big XML document if you don't have enough resources. I would suggest reformatting this file. For example, I would parse this XML and insert data into a SQL database. Then, you can easily operate with such amounts of data.

Answer 3

Why do you need to convert it to a string?

using System.IO;

public static void WriteBytes(byte[] bytes, string filename)
{
    using (FileStream fs = new FileStream(filename, FileMode.OpenOrCreate))
    using (BinaryWriter writer = new BinaryWriter(fs, Encoding.UTF8))
    {
        writer.Write(bytes);
    }
}

Answer 4

I would prefer that I write all bytes to file. And when reading, convert it to string and then convert to XML using XDocument, XElement etc. By writing bytes in file you will save space, and it is efficient,

Instead of using FileStream, I will prefer File.WriteAllBytes method.

private const string fileName = "ServerData.xml";
public void ProcessBuffer(byte[] receiveBuffer, int bytes)
{
    File.WriteAllBytes(filename, bytes);


    // And when reading
    var bytes = File.ReadAllBytes(filename);
    var binaryReader = new BinaryReader(new MemoryStream(bytes));
    // Parse strings and make xml,
    binaryReader.ReadString();

}