String Ansi to UTF-8

Question

Have some problem. Have large txt file in ANsi.

Read it line by line with such function:

private static IEnumerable<string> ReadLineFromFile(TextReader fileReader)
{
    using (fileReader)
    {
        string currentLine;
        while ((currentLine = fileReader.ReadLine()) != null)
        {
            yield return currentLine;
        }
    }
}


public void go()
{
    while (true)
    {
        TextReader readFile = new StreamReader(file_path);
        foreach (string line in ReadLineFromFile(readFile))
        {
        }
    }
}

How to convert all ANSI lines to UTF-8? Thanks

http://stackoverflow.com/questions/3778241/converting-problem-ansi-to-utf8-c-sharp does this solve your problem? — Fabian Bigler, Jun 04 '13 at 21:45
You can use File.ReadLines() with a parameter of Encoding.Default — Matthew Watson, Jun 04 '13 at 22:29

score 0 · Answer 1 · answered Jun 04 '13 at 21:47

Try using Encoding.UTF8.GetBytes() (in the System.Text namespace) to get the bytes that correspond to a UTF-8 string. In .NET, all strings are internally Unicode, so there's no such thing as a UTF-8 string at runtime. Rather, when encoding strings in different formats, you'll use methods like GetBytes() to get the bytes that represent the encoded string in memory.

EDIT: Some links:

score 0 · Answer 2 · answered Jun 04 '13 at 21:59

First you need to read the bytes from the file, then use Encoding.GetEncoding(1252); to get the ANSI encoding (the code page may vary), then you can use GetString to get an internal .net-string or convert it to another encoding.

Try something like this:

private IEnumerable<string> ReadLineFromFile(string path)
{
    byte[] ansiEncodedBytes = File.ReadAllBytes(path);
    Encoding ansi = Encoding.GetEncoding(1252);
    string utf16string = ansi.GetString(ansiEncodedBytes);
    return utf16string.Split("\n");
}

score 0 · Answer 3 · answered Jun 05 '13 at 06:21

If you are using .Net 4 or later, you can use the File.ReadLines(string path, Encoding encoding) method.

This reads the file line-by-line like your ReadLineFromFile() method, and the Encoding parameter will allow you to specify Encoding.Default. This will tell it to use the operating system's current ANSI code page when reading the text.

Note that the strings will be converted from ANSI to UTF16, because UTF16 is the type used for string in C#.

So you could rewrite your go() test method like so:

using System.IO;
using System.Text;

...

public void go()
{
    while (true)
    {
        foreach (string line in File.ReadLines(file_path, Encoding.Default))
        {
        }
    }
}

String Ansi to UTF-8

3 Answers3