Read text encoding issue

Question

I have read a certain plain text file (csv) and I have a problem with xA0

Visual Studio 2015:

Notepad++: (when setting char encoding to utf-8)

so it seems to be this non-breaking space so I tried this:

temp = temp.Replace("\xA0", string.Empty);

But it did not work and gave me the black squares similar to the first screenshot. I also changed

System.IO.StreamReader sr = new System.IO.StreamReader(csvFile.FileContent);

to use specific utf-8 encoding:

System.IO.StreamReader sr = new System.IO.StreamReader(csvFile.FileContent, System.Text.Encoding.UTF8);

both gave the same result. I really dislike char encoding and could use some help and explanation about my mistake.

edit added the notepad++ hex view: (to confirm it is the non-breaking char)

edit2 changing the streamreader constructor values to this:

System.IO.StreamReader sr = new System.IO.StreamReader(csvFile.FileContent, true);

results in an utf-8 encoding for reading the file. I tried to convert the latin1 to utf-8 but that gave me ??? https://stackoverflow.com/a/13999801/169714

Encoding.UTF8.GetString(Encoding.GetEncoding("iso-8859-1").GetBytes(temp))

Answer 1

try putting the result into a string, reading the data and printing out the result

something like this:

string[] data = File.ReadAllLines(yourSavePath); 
File.WriteAllLines(yourSavePath, data);

if i am right it should fix it, it is a missing characters issue

Answer 2

0xA0 is the non-breaking space in Latin1 , iso-8859-1. You can read it by passing Encoding.GetEncoding("iso-8859-1") as the encoding:

var latin1= Encoding.GetEncoding("iso-8859-1");
var sr = new System.IO.StreamReader(csvFile.FileContent, latin1);

For example, for the input array:

byte[] values={0x53,0x34,0x35,0x3b,0x35,0x31,0xa0,0xa0,0xa0,0xa0,0xa0};

UTF8 returns

var s1=Encoding.UTF8.GetString(values);
Console.WriteLine(s1);

S45;51�����

While Latin1 returns a valid string

var s2=latin1.GetString(values);
Console.WriteLine(s2);

S45;51

.NET uses Unicode for strings and text files are read using UTF8 by default. Eg, StreamReader's constructor defaults to UTF8:

    public StreamReader(Stream stream) 
        : this(stream, true) {
    }

    public StreamReader(Stream stream, bool detectEncodingFromByteOrderMarks) 
        : this(stream, Encoding.UTF8, detectEncodingFromByteOrderMarks, DefaultBufferSize, false) {
    }

To use the system locale, the Encoding.Default encoding must be passed explicitly.

var sr = new System.IO.StreamReader(csvFile.FileContent, Encoding.Default);

Many West European and English-speaking countries do use this encoding, so the system locale could be expected to be Latin1. This is a risky assumption to make in import jobs though