Thank you all for some very useful answers. I realize the actual question isn't "How can I convert ANY Unicode character into its ASCII fallback" - the question is "how can I convert the Unicode characters my customers are complaining about into their ASCII fallbacks" ?
In other words - we don't need a general-purpose solution; we need a solution that'll work 99% of the time, for English-speaking customers pasting English-language content from Word and other websites into our application. To that end, I analyzed eight years' worth of messages sent through our system looking for characters that aren't representable in ASCII encoding, using this test:
///<summary>Determine whether the supplied character is
///using ASCII encoding.</summary>
bool IsAscii(char inputChar) {
var ascii = new ASCIIEncoding();
var asciiChar = (char)(ascii.GetBytes(inputChar.ToString())[0]);
return(asciiChar == inputChar);
}
I've then been through the resulting set of unrepresentable characters and manually assigned an appropriate replacement string. The whole lot is bundled up in an extension method, so you can call myString.Asciify() to convert your string into a reasonable ASCII-encoding approximation.
public static class StringExtensions {
private static readonly Dictionary<char, string> Replacements = new Dictionary<char, string>();
/// <summary>Returns the specified string with characters not representable in ASCII codepage 437 converted to a suitable representative equivalent. Yes, this is lossy.</summary>
/// <param name="s">A string.</param>
/// <returns>The supplied string, with smart quotes, fractions, accents and punctuation marks 'normalized' to ASCII equivalents.</returns>
/// <remarks>This method is lossy. It's a bit of a hack that we use to get clean ASCII text for sending to downlevel e-mail clients.</remarks>
public static string Asciify(this string s) {
return (String.Join(String.Empty, s.Select(c => Asciify(c)).ToArray()));
}
private static string Asciify(char x) {
return Replacements.ContainsKey(x) ? (Replacements[x]) : (x.ToString());
}
static StringExtensions() {
Replacements['’'] = "'"; // 75151 occurrences
Replacements['–'] = "-"; // 23018 occurrences
Replacements['‘'] = "'"; // 9783 occurrences
Replacements['”'] = "\""; // 6938 occurrences
Replacements['“'] = "\""; // 6165 occurrences
Replacements['…'] = "..."; // 5547 occurrences
Replacements['£'] = "GBP"; // 3993 occurrences
Replacements['•'] = "*"; // 2371 occurrences
Replacements[' '] = " "; // 1529 occurrences
Replacements['é'] = "e"; // 878 occurrences
Replacements['ï'] = "i"; // 328 occurrences
Replacements['´'] = "'"; // 226 occurrences
Replacements['—'] = "-"; // 133 occurrences
Replacements['·'] = "*"; // 132 occurrences
Replacements['„'] = "\""; // 102 occurrences
Replacements['€'] = "EUR"; // 95 occurrences
Replacements['®'] = "(R)"; // 91 occurrences
Replacements['¹'] = "(1)"; // 80 occurrences
Replacements['«'] = "\""; // 79 occurrences
Replacements['è'] = "e"; // 79 occurrences
Replacements['á'] = "a"; // 55 occurrences
Replacements['™'] = "TM"; // 54 occurrences
Replacements['»'] = "\""; // 52 occurrences
Replacements['ç'] = "c"; // 52 occurrences
Replacements['½'] = "1/2"; // 48 occurrences
Replacements[''] = "-"; // 39 occurrences
Replacements['°'] = " degrees "; // 33 occurrences
Replacements['ä'] = "a"; // 33 occurrences
Replacements['É'] = "E"; // 31 occurrences
Replacements['‚'] = ","; // 31 occurrences
Replacements['ü'] = "u"; // 30 occurrences
Replacements['í'] = "i"; // 28 occurrences
Replacements['ë'] = "e"; // 26 occurrences
Replacements['ö'] = "o"; // 19 occurrences
Replacements['à'] = "a"; // 19 occurrences
Replacements['¬'] = " "; // 17 occurrences
Replacements['ó'] = "o"; // 15 occurrences
Replacements['â'] = "a"; // 13 occurrences
Replacements['ñ'] = "n"; // 13 occurrences
Replacements['ô'] = "o"; // 10 occurrences
Replacements['¨'] = ""; // 10 occurrences
Replacements['å'] = "a"; // 8 occurrences
Replacements['ã'] = "a"; // 8 occurrences
Replacements['ˆ'] = ""; // 8 occurrences
Replacements['©'] = "(c)"; // 6 occurrences
Replacements['Ä'] = "A"; // 6 occurrences
Replacements['Ï'] = "I"; // 5 occurrences
Replacements['ò'] = "o"; // 5 occurrences
Replacements['ê'] = "e"; // 5 occurrences
Replacements['î'] = "i"; // 5 occurrences
Replacements['Ü'] = "U"; // 5 occurrences
Replacements['Á'] = "A"; // 5 occurrences
Replacements['ß'] = "ss"; // 4 occurrences
Replacements['¾'] = "3/4"; // 4 occurrences
Replacements['È'] = "E"; // 4 occurrences
Replacements['¼'] = "1/4"; // 3 occurrences
Replacements['†'] = "+"; // 3 occurrences
Replacements['³'] = "'"; // 3 occurrences
Replacements['²'] = "'"; // 3 occurrences
Replacements['Ø'] = "O"; // 2 occurrences
Replacements['¸'] = ","; // 2 occurrences
Replacements['Ë'] = "E"; // 2 occurrences
Replacements['ú'] = "u"; // 2 occurrences
Replacements['Ö'] = "O"; // 2 occurrences
Replacements['û'] = "u"; // 2 occurrences
Replacements['Ú'] = "U"; // 2 occurrences
Replacements['Œ'] = "Oe"; // 2 occurrences
Replacements['º'] = "?"; // 1 occurrences
Replacements['‰'] = "0/00"; // 1 occurrences
Replacements['Å'] = "A"; // 1 occurrences
Replacements['ø'] = "o"; // 1 occurrences
Replacements['˜'] = "~"; // 1 occurrences
Replacements['æ'] = "ae"; // 1 occurrences
Replacements['ù'] = "u"; // 1 occurrences
Replacements['‹'] = "<"; // 1 occurrences
Replacements['±'] = "+/-"; // 1 occurrences
}
}
Note that there are some rather odd fallbacks in there - like this one:
Replacements['³'] = "'"; // 3 occurrences
Replacements['²'] = "'"; // 3 occurrences
That's because one of our users has some program that converts open/close smart-quotes into ² and ³ (like : he said ²hello³) and nobody has ever used them to represent exponentiation, so this will probably work quite nicely for us, but YMMV.
