How to read text file without knowing the encoding

Question

When reading a text file that was created somewhere else outside my app, the encoding used is unknown. My app has being using NSUnicodeStringEncoding (which is the same as NSUTF16StringEncoding) so have problems reading other than UTF16 encoded files.

Is there a way I can guess the encoding of a file? My priority is to be able to read UTF8 files and then all other files. Is iterating through available encodings and check if read string's length is more than zero is really a good approach?

Thanks in advance.

Ignacio

There are a few related questions on Stack Overflow: http://stackoverflow.com/questions/1351151/guess-encoding-when-creating-an-nsstring-from-nsdata http://stackoverflow.com/questions/4198804/how-to-reliably-guess-the-encoding-between-macroman-cp1252-latin1-utf-8-and-a http://stackoverflow.com/questions/373081/how-can-i-best-guess-the-encoding-when-the-bom-byte-order-mark-is-missing (and others). You could adapt chardet to Objective-C: http://chardet.feedparser.org/ — , Feb 09 '11 at 01:54

score 10 · Accepted Answer · edited Feb 09 '11 at 01:54

Apple's documentation has some guidance on how to proceed: String Programming Guide: Reading data with an unknown encoding:

If you are forced to guess the encoding (and note that in the absence of explicit information, it is a guess):

Try stringWithContentsOfFile:usedEncoding:error: or initWithContentsOfFile:usedEncoding:error: (or the URL-based equivalents). These methods try to determine the encoding of the resource, and if successful return by reference the encoding used.

If (1) fails, try to read the resource by specifying UTF-8 as the encoding.

If (2) fails, try an appropriate legacy encoding. "Appropriate" here depends a bit on circumstances; it might be the default C string encoding, it might be ISO or Windows Latin 1, or something else, depending on where your data is coming from.

score 1 · Answer 2 · answered Feb 09 '11 at 02:02

1

If the file is properly constructed you can read the first four bytes and see if it is a BOM (Byte Order Mark):

http://en.wikipedia.org/wiki/Byte-order_mark

answered Feb 09 '11 at 02:02

Steve Wellens

20,506
2
28
69

Not very useful. That only tells you the endianess of a Unicode encoding. – Raedwald Feb 10 '12 at 17:33
It also tells you the encoding: UTF-8, UTF-16 (BE), UTF-16 (LE), UTF-32 (BE), UTF-32 (LE), UTF-7, UTF-EBCDIC, etc. – Steve Wellens Feb 10 '12 at 17:59

How to read text file without knowing the encoding

2 Answers2

Linked