Unicode Encoding - handling Byte Order Mark

Question

I need to serialise some objects to a blob to pass in to a web service call. All pretty basic stuff.

The nub of the problem is that the snippet of code that does the work is pretty inflexible in that when using a StringWriter, the encoding of the output is always UTF-16

        StringWriter stringWriter = new StringWriter();
        serialiser.Serialize(stringWriter, Container.Calls);
        string data = stringWriter.ToString();

That handles most scenarios but I want to make this as generic as possible & deal with other encodings so I changed the above to this (with the idea that I could refactor and pass in the encoding at a later time):

        XmlTextWriter xmlTextWriter = new XmlTextWriter(stream,Encoding.Unicode);
        serialiser.Serialize(xmlTextWriter, Container.Calls);
        byte[] bytes = stream.GetBuffer();
        string data = System.Text.Encoding.Unicode.GetString(bytes);

System.Text.Encoding.Unicode.GetString no longer returns valid xml as there is a Byte Order Mark at the start of the stream. I could call stream.Read and specify the offset of the BOM but, dependinmg on the encoding, I don't always expect it to be present so this could get messy. What I really need is for it not to be there at all.

http://msdn.microsoft.com/en-us/library/984ebfcy%28v=vs.110%29.aspx — Hans Passant, May 20 '14 at 10:36

score 1 · Accepted Answer · edited May 23 '17 at 10:26

From another post (StreamWriter and UTF-8 Byte Order Marks):

"The issue is due to the fact that you are using the static UTF8 property on the Encoding class.

When the GetPreamble method is called on the instance of the Encoding class returned by the UTF8 property, it returns the byte order mark (the byte array of three characters) and is written to the stream before any other content is written to the stream"

So, in this case I change

XmlTextWriter xmlTextWriter = new XmlTextWriter(stream,Encoding.Unicode);

To

XmlTextWriter xmlTextWriter = new XmlTextWriter(stream,new System.Text.UnicodeEncoding  (false,false));

and it works ok.

Unicode Encoding - handling Byte Order Mark

1 Answers1