Finding the encoding of text files

Question

I have a bunch of text files with different encodings. But I want to convert all of the into utf-8. since there are about 1000 files, I cant do it manually. I know that there are some commands in llinux which change the encodings of files from one encoding into another one. but my question is how to automatically detect the current encoding of a file? Clearly I'm looking for a command (say FindEncoding($File) ) to do this:

foreach file
do
$encoding=FindEncoding($File);
uconv -f $encoding -t utf-8 $file;
done

`enca` seems to be the tool you are looking for: http://linux.die.net/man/1/enca — Anders Lindahl, Oct 12 '12 at 19:59
`uconv` should detect the source encoding automatically, i.e. you shouldn't need the `-f` flag. — Lars Kotthoff, Oct 13 '12 at 00:52
possible duplicate of [How to find encoding of a file in Unix via script(s)](http://stackoverflow.com/questions/805418/how-to-find-encoding-of-a-file-in-unix-via-scripts) — Jonas Stein, Mar 15 '14 at 22:29

score 5 · Answer 1 · answered Jan 08 '13 at 22:54

I usually do sth like this:

for f in *.txt; do
    encoding=$(file -i "$f" | sed "s/.*charset=\(.*\)$/\1/")
    recode $encoding..utf-8 "$f"
done

Note that recode will overwrite the file for changing the character encoding. If it is not possible to identify the text files by extension, their respective mime type can be determined with file -bi | cut -d ';' -f 1.

It is also probably a good idea to avoid unnecessary re-encodings by checking on UFT-8 first:

if [ ! "$encoding" = "utf-8" ]; then
    #encode

After this treatment, there might still be some files with an us-ascii encoding. The reason for that is ASCII being a subset of UTF-8 that remains in use unless any characters are introduced that are not expressible by ASCII. In that case, the encoding switches to UTF-8.

Finding the encoding of text files

1 Answers1