How do I handle non-English characters properly?

Question

So I'm working with last.fm API. Sometimes, the query results in tracks that contain characters like these:

Æther, é, Hṛṣṭa

or non-English characters like these:

水鏡.

When debugging in Eclipse, I see them just fine (as-is) but printing on console prints these as ??? - which is OK for me.

Now, how do I handle these? At first I though I could remove every song that has any character other than the ones in English language. I used the regex ^\\w+$ but it didn't work. I also tried \\w+. That didn't work either.

Then I thought further on how do handle these properly. Any one can help me out? I am perfectly fine with letting these tracks out of the equation, ie. I'm fine with having only English character tracks.

Another question: What is the best way to display these character of console and/or Swing GUI?

The problem is likely that the console does not support UTF-8 or the font it is using doesn't have glyphs for them. — Sled, Mar 27 '13 at 20:50
Do you mean the Eclipse output console, or an actual operating system command line console? If command line, which operating system and console are you using? — Bobulous, Mar 27 '13 at 20:52
Ah, so I changed the encoding to UTF-8. Perfect. So now, how can I check whether a string contains such characters? — Karan Goel, Mar 27 '13 at 20:53
is this what you want [Regular expression to match non-english characters?](http://stackoverflow.com/questions/150033/regular-expression-to-match-non-english-characters) — BevynQ, Mar 27 '13 at 21:02
@BevynQ Perfect! Work. But could you explain what the `x80` means? The last hex code should be `7F` right? — Karan Goel, Mar 27 '13 at 21:10
"At first I though I could remove every song that has any character other than the ones in English language." WHYYYYYYYYYYYY — R. Martinho Fernandes, Mar 28 '13 at 13:17
@PavelRadzivilovsky But I'm just concerned what if some users do not have support for UTF-8 and the text messes up? — Karan Goel, Mar 28 '13 at 14:43
We are in the year 2013 - everybody should sufficiently support UTF-8. — Pavel Radzivilovsky, Mar 28 '13 at 17:46

score 0 · Answer 1 · answered Mar 27 '13 at 20:54

0

You must ensure that you use correct encoding when reading your input first.

Second ensure that the font used in Eclipse on platform you developing has ability to display all these characters. Swing must display unicode chars if you read them correctly.

You will likely want to use UTF-8 everywhere.

answered Mar 27 '13 at 20:54

Vladimir

4,782
7
35
56

1

So if I have the correct encoding on my development platform, is it guaranteed that the program will work fine on other platforms as well? – Karan Goel Mar 27 '13 at 20:56
If you give your correct app to another user he will need to use correct font as well on his platform in console. E.g. on Linux some default fonts could be not Unicode. – Vladimir Mar 27 '13 at 21:01
so basically there are three issues with encoding: 1. how you read your input. 2. how you write output and 3. fonts which display output :-) – Vladimir Mar 27 '13 at 21:02

How do I handle non-English characters properly?

1 Answers1