C# - Detecting encoding in a file, write change to file using the found encoding

Question

I wrote a small program for iterating through a lot of files and applying some changes where a certain string match is found, the problem I have is that different files have different encodings. So what I would like to do is check the encoding, then overwrite the file in its original encoding.

What would be the prettiest way of doing that in C# .net 2.0?

My code looks very simple as of now;

String f1 = File.ReadAllText(fileList[i]).ToLower();

if (f1.Contains(oPath))
{
    f1 = f1.Replace(oPath, nPath);
    File.WriteAllText(fileList[i], f1, Encoding.Unicode);
}

I took a look at Auto encoding detect in C# which made me realize how I could detect encoding, but I am not sure how I could use that information to write in the same encoding.

Would greatly appreciate any help here.

Read this http://www.joelonsoftware.com/articles/Unicode.html — RvdK, Dec 08 '10 at 09:33
Possibly related: https://stackoverflow.com/questions/48380031/detect-the-encoding-of-a-text-file-using-c-sharp — Missy, Apr 06 '22 at 16:55

JaredPar · Accepted Answer · 2010-12-08T09:17:29.650

Unfortunately encoding is one of those subjects where there is not always a definitive answer. In many cases it's much closer to guessing the encoding as opposed to detecting it. Raymond Chen did an excellent blog post on this subject that is worth the read

http://blogs.msdn.com/b/oldnewthing/archive/2007/04/17/2158334.aspx

The gist of the article is

If the BOM (byte order marker) exists then you're golden
Else it's guess work and heuristics

However I still think the best approach is to Darin mentioned in the question you linked. Let StreamReader guess for you vs. re-inventing the wheel. It only requires a very slight modification to your sample.

String f1;
Encoding encoding;
using (var reader = new StreamReader(fileList[i])) {
  f1 = reader.ReadToEnd().ToLower();
  encoding = reader.CurrentEncoding;
}

if (f1.Contains(oPath))
{
  f1 = f1.Replace(oPath, nPath);
  File.WriteAllText(fileList[i], f1, encoding);
}

Ahh, I just learned something vital, thank you for that. Good answer. — cc0, Dec 08 '10 at 09:14
+1. But in my case, I had to swap `WriteAllText` to `WriteAllBytes`, as in `File.WriteAllBytes(fileList[i], encoding.GetBytes(f1));`, otherwise in cases where the encoding was UTF-8, the file ended up with a BOM preamble, whereas the original file didn't have it, and the resulting file shouldn't as well. — Marcos Dimitrio, Sep 04 '20 at 20:39

score 2 · Answer 2 · answered Dec 08 '10 at 09:10

2

By default, .Net use UTF8. It is hard to detect character encoding becus most of the time .Net will read as UTF8. i alway have problem with ANSI.

my trick is i will read the file as Stream as force it to read as UTF8 and detect usual character that should be in text. If found, then UTF8 else ANSI ... and tell user u can use just 2 encoding either ANSI or UTF8. auto dectect not quite work in my language :p

answered Dec 08 '10 at 09:10

Bonshington

3,970
2
25
20

Good advice, in the most of cases, for text files the encoding is or the Default (Windows) or UTF8. – dlopezgonzalez Nov 13 '14 at 11:36

score 1 · Answer 3 · answered Sep 10 '13 at 13:25

Prob a bit late but I encountered the same problem myself, using the previous answers I found a solution that works for me, It reads in the text using StreamReaders default encoding, extracts the encoding used on that file and uses StreamWriter to write it back with the changes using the found Encoding. Also removes\reAdds the ReadOnly flag

        string file = "File to open";
        string text;
        Encoding encoding;
        string oldValue = "string to be replaced";
        string replacementValue = "New string";

        var attributes = File.GetAttributes(file);
        File.SetAttributes(file, attributes & ~FileAttributes.ReadOnly);

        using (StreamReader reader = new StreamReader(file, Encoding.Default))
        {
            text = reader.ReadToEnd();
            encoding = reader.CurrentEncoding;
            reader.Close();
        }

        bool changedValue = false;
        if (text.Contains(oldValue))
        {
            text = text.Replace(oldValue, replacementValue);
            changedValue = true;
        }

        if (changedValue)
        {
            using (StreamWriter write = new StreamWriter(file, false, encoding))
            {
                write.Write(text.ToString());
                write.Close();
            }
            File.SetAttributes(file, attributes | FileAttributes.ReadOnly);
        }

score 1 · Answer 4 · edited May 23 '17 at 12:04

I am afraid, you will have to know the encoding. For UTF based encodings though you can use StreamReader built in functionality though.

Taken form here.

With regard to encodings - you will need to have identified the encoding in order to use the StreamReader.

However, the StreamReader itself can help if you create it with one of the constructor overloads that allows you to supply the flag detectEncodingFromByteOrderMarks as true (or you can use Encoding.GetPreamble and look at the byte preamble yourself).

Both these methods will only help auto-detect UTF based encodings though - so any ANSI encodings with a specified codepage will probably not be parsed correctly.

Michael Hutter · Answer 5 · 2021-09-24T09:27:58.807

0

The solution for all Germans => ÄÖÜäöüß

This function opens the file an determines the Encoding by the BOM.
If the BOM is missing the file will be interpreted as ANSI, but if there are UTF8 encoded German Umlaute in it, it will be detected as UTF8.

https://stackoverflow.com/a/69312696/9134997

edited Sep 24 '21 at 09:27

answered Sep 24 '21 at 07:58

Michael Hutter

1,064
13
33

C# - Detecting encoding in a file, write change to file using the found encoding

5 Answers5