I know the web is mostly standardizing towards UTF-8 lately and I was just wondering if there was any place where using UTF-8 would be a bad thing. I've heard the argument that UTF-8, 16, etc may use more space but in the end it has been negligible.
Also, what about in Windows programs, Linux shell and things of that nature -- can you safely use UTF-8 there?
When you need to write a program (performing string manipulations) that needs to be very very fast and that you're sure that you won't need exotic characters, may be UTF-8 is not the best idea. In every other situations, UTF-8 should be a standard.
UTF-8 works well on almost every recent software, even on Windows.
If UTF-32 is available, prefer that over the other versions for processing.
If your platform supports UTF-32/UCS-4 Unicode natively - then the "compressed" versions UTF-8 and UTF-16 may be slower, because they use varying numbers of bytes for each character (character sequences), which makes impossible to do a direct lookup in a string by index, while UTF-32 uses 32 bit "flat" for each character, speeding up some string operations a lot.
Of course, if you are programming in a very restricted environment like, say, embedded systems and can be certain there will be only ASCII or ISO 8859-x characters around, ever, then you can chose those charsets for efficiency and speed. But in general, stick with the Unicode Transformation Formats.
I know the web is mostly standardizing towards UTF-8 lately and I was just wondering if there was any place where using UTF-8 would be a bad thing.
There is an argument to be made that adding unnecessary conversions is adding complexity for little benefit. So if your inputs and your outputs use the same format then there is an argument for working in that format too.
Both UTF-8 and UTF-16 are relatively well-designed multi-unit encodings. A smaller sequence of code units never appears as a sub-sequence of a longer sequence and a decoder that detects an error can resume decoding at the next valid code unit.
Some argue that UTF-32 is "better" because it uses one code unit for every Unicode code point. What makes this more questionable though is that there is not a 1:1 mapping between unicode code points and what most users would regard as "characters". So being able to rapidly get the nth code point from a sequence is less useful than it would first appear.
Also, what about in Windows programs, Linux shell and things of that nature -- can you safely use UTF-8 there?
Windows and Unix-like systems took different approaches to the introduction of Unicode. Both approaches had their pros and cons.
Windows introduced 16 bit Unicode (initially UCS-2, later UTF-16) by introducing a parallel set of APIs. Applications or frameworks that wanted Unicode support had to switch to the new APIs. This was further complicated by the fact that while windows NT offered Unicode support in all APIs, windows 9x only offered it in a subset.
On the filesystem side, windows NT's native NTFS filesystem used 16 bit unicode filenames from the start. For the FAT filesystem which pre-dated windows NT, Unicode was introduced as part of Long filename support. Similarly for CDs, the Joliet extension added Unicode long filenames.
So did the long filename extensions for FAT, and the Joliet long filename extensions for CDs.
Unix-like systems on the other hand introduced Unicode by using UTF-8 and treating it like any other extended-ascii character set. Filenames on Unix filesystems have always been sequences of bytes, where the meaning assigned to those bytes is down to the user's environment.
There are pros and cons to both approaches. The Unix approach allowed even non unicode aware programs to handle Unicode text to some extent. On the other hand it meant users had to essentially choose between a "Unicode" environment where everything was UTF-8 and where any pre-unicode files would need conversion and a "legacy" environment where Unicode was not supported.
Some programming languages or frameworks will attempt to settle on an encoding and convert everything to that encoding. This is however complicated by the fact that on both Windows and Unix-like systems a program may encounter strings from the operating system that do not pass validation for their nominal encoding. This can happen for a number of reasons, including legacy data from pre-transition software, truncation that does not take account of the multi-unit encodings and use of what are nominally text strings to pass non-text data and just plain old errors.
It is well-known that utf-8 works best for file storage and network transport. But people debate whether utf-16/32 are better for processing. One major argument is that utf-16 is still variable length and even utf-32 is still not one code-point per character, so how are they better than utf-8? My opinion is that utf-16 is a very good compromise.
First, characters out side of BMP which need double code-points in utf-16 are extremely rarely used ones. The Chinese characters (also some other Asia characters) in that range are basically dead ones. Ordinary people won't use them at all, except experts use them to digitalize ancient books. So, utf-32 will be a waste most of the time. Don't worry too much about those characters, as they won't make your software look bad if you didn't handle them properly, as long as your software is not for those special users.
Second, often we need the string memory allocation to be related to character count. e.g. a database string column for 10 characters (assuming we store unicode string in normalized form), which will be 20 bytes for utf-16. In most cases it will work just like that, except in extreme cases it will hold only 5-8 characters. But for utf-8, the common byte length of one character is 1-3 for western languages and 3-5 for Asia languages. Which means we need 10-50 bytes even for the common cases. More data, more processing.
