read unicode string from text file in UWP app

Question

in Windows 10 app I try to read string from .txt file and set the text to RichEditBox:

Code variant 1:

var read = await FileIO.ReadTextAsync(file, Windows.Storage.Streams.UnicodeEncoding.Utf8);
txt.Document.SetText(Windows.UI.Text.TextSetOptions.None, read);

Code variant 2:

var stream = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite);
ulong size = stream.Size;
using (var inputStream = stream.GetInputStreamAt(0))
{
    using (var dataReader = new Windows.Storage.Streams.DataReader(inputStream))
    {
        dataReader.UnicodeEncoding = Windows.Storage.Streams.UnicodeEncoding.Utf8;
        uint numBytesLoaded = await dataReader.LoadAsync((uint)size);
        string text = dataReader.ReadString(numBytesLoaded);
        txt.Document.SetText(Windows.UI.Text.TextSetOptions.FormatRtf, text);
    }
}

On some files I have this error - "No mapping for the Unicode character exists in the target multi-byte code page"

I found one solution:

IBuffer buffer = await FileIO.ReadBufferAsync(file);
DataReader reader = DataReader.FromBuffer(buffer);
byte[] fileContent = new byte[reader.UnconsumedBufferLength];
reader.ReadBytes(fileContent);
string text = Encoding.UTF8.GetString(fileContent, 0, fileContent.Length);
txt.Document.SetText(Windows.UI.Text.TextSetOptions.None, text);

But with this code the text looks like question marks in rhombus.

How I can read and display same text files in normal encoding?

Are you sure, your text is UTF8 encoded? – TGlatzer Feb 09 '16 at 16:49 — TGlatzer, Feb 09 '16 at 16:49
No. Its Unicode string in codepage 1251 or 1252. As I think – SYL Feb 09 '16 at 21:24 — SYL, Feb 09 '16 at 21:24

score 3 · Answer 1 · edited May 23 '17 at 12:02

Challenge here is the encoding and it depends how much accuracy you need for your application. If you need something fast and simple you can adapt this answer

    public static Encoding GetEncoding(byte[4] bom)
    {
        // Analyze the BOM
        if (bom[0] == 0x2b && bom[1] == 0x2f && bom[2] == 0x76) return Encoding.UTF7;
        if (bom[0] == 0xef && bom[1] == 0xbb && bom[2] == 0xbf) return Encoding.UTF8;
        if (bom[0] == 0xff && bom[1] == 0xfe) return Encoding.Unicode; //UTF-16LE
        if (bom[0] == 0xfe && bom[1] == 0xff) return Encoding.BigEndianUnicode; //UTF-16BE
        if (bom[0] == 0 && bom[1] == 0 && bom[2] == 0xfe && bom[3] == 0xff) return Encoding.UTF32;
        return Encoding.ASCII;
    }

    async System.Threading.Tasks.Task MyMethod()
    {
        FileOpenPicker openPicker = new FileOpenPicker();
        StorageFile file = await openPicker.PickSingleFileAsync();
        IBuffer buffer = await FileIO.ReadBufferAsync(file);
        DataReader reader = DataReader.FromBuffer(buffer);
        byte[] fileContent = new byte[reader.UnconsumedBufferLength];
        reader.ReadBytes(fileContent);
        string text = GetEncoding(new byte[4] {fileContent[0], fileContent[1], fileContent[2], fileContent[3] }).GetString(fileContent);
        txt.Document.SetText(Windows.UI.Text.TextSetOptions.None, text);

        //.. 
    }

If you need something more accurate you should think to port to UWP a porting to .Net of Mozilla charset detector as already mentioned in this answer

Please note that the code above is just a sample it is missing all the using statements for types implementing IDisposable and it also should have been wrote in a more consistent way

hth -g

Very thx. I has made UWP port of Mozilla charset detector, as you say. And use "Portable.Text.Encoding" library from NuGet. It works perfect ) — SYL, Feb 10 '16 at 11:22
I use the notepad save as ascii and I can't open it when use your code ,and all text is "?" when char is chinese. — lindexi, Jul 12 '16 at 13:51

score 0 · Accepted Answer · answered Feb 12 '16 at 12:00

Solution:

1) I made a port of Mozilla Universal Charset Detector to UWP (added to Nuget)

ICharsetDetector cdet = new CharsetDetector();
cdet.Feed(fileContent, 0, fileContent.Length);
cdet.DataEnd();

2) Nuget library Portable.Text.Encoding

if (cdet.Charset != null)
string text = Portable.Text.Encoding.GetEncoding(cdet.Charset).GetString(fileContent, 0, fileContent.Length);

That's all. Now unicode ecnodings (include cp1251, cp1252) - works good ))

score 0 · Answer 3 · edited Jul 12 '16 at 14:28

        StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets/FontFiles/" + fileName));
        using (var inputStream = await file.OpenReadAsync())
        using (var classicStream = inputStream.AsStreamForRead())
        using (var streamReader = new StreamReader(classicStream))
        {
            while (streamReader.Peek() >= 0)
            {
                line = streamReader.ReadLine();
           }
       }

read unicode string from text file in UWP app

3 Answers3