Chinese Strings Handling in Java?

Question

In my assigned project, the original author has written a function:

  public String asString() throws DataException
  {
    if (getData() == null) return null;

    CharBuffer charBuf = null;

    try
    {
        charBuf = s_charset.newDecoder().decode(ByteBuffer.wrap(f_data));
    }
    catch (CharacterCodingException e)
    {
        throw new DataException("You can't have a string from this ParasolBlob: " + this, e);
    }

    return charBuf.toString()+"你好";
 }

Please note that the constant s_charset is defined as:

private static final Charset s_charset = Charset.forName("UTF-8");

Please also note that I have hard-coded a Chinese string in the return string.

Now when the program flow reaches this method, it will throw the following exception:

 java.nio.charset.UnmappableCharacterException: Input length = 2

And more interstingly, the hard-coded Chinese strings will be shown as "??" at the console if I do a System.out.println().

I think this problem is quite interesting in regard of Localization. And I've tried changing it to Charset.forName("GBK");

but seems is not the solution. Also, I have set the coding of the Java class to be of "UTF-8".

Any experts have experience in this regard? Would you please share a little? Thanks in advance!

Juned Ahsan · Accepted Answer · 2013-07-15T10:15:28.163

3

And more interstingly, the hard-coded Chinese strings will be shown as "??" at the console if I do a System.out.println().

System.out performs transcoding operations from UTF-16 strings to the default JRE character encoding. If this does not match the encoding used by the device receiving the character data is corrupted. So, the console should be set to use the right character encoding(UTF-8) to render the chinese chars properly.

If you are using eclipse then you can change the console encoding by going to

Run Configuration-> Common -> Encoding(slect UTF-8 from dropdown)

enter image description here

edited Jul 15 '13 at 10:15

answered Jul 15 '13 at 09:55

Juned Ahsan

67,789
12
98
136

+1, however I am not sure that OP mean console of eclipse. If you are using for example windows it is more complicated. But this discussion will help you: http://stackoverflow.com/questions/388490/unicode-characters-in-windows-command-line-how – AlexR Jul 15 '13 at 09:57
It works well on my console now, thank you. But my ultimate goal is to extract the strings and output it to a csv file. It is still shown as ??? in the csv file. I already set the encoding for the csv. – Kevin Jul 16 '13 at 02:00
@Kevin Read [this](http://stackoverflow.com/a/16436195/1163607). – AllTooSir Jul 16 '13 at 04:27

score 0 · Answer 2 · answered Jul 15 '13 at 09:58

0

Java Strings are unicodes

System.out.println("你好");

answered Jul 15 '13 at 09:58

Ruchira Gayan Ranaweera

34,993
17
75
115

score 0 · Answer 3 · answered Jul 15 '13 at 17:53

As Kevin stated, depending on what is the underlying encoding of your source file, this encoding will be used to convert it to UTF-16BE (real encoding of Java String). So when you see "??" it is surely simple conversion error.

Now, if you want to convert simple byte array to String, using given character encoding, I believe there is much easier way to do this, than using raw CharsetDecoder. That is:

byte[] bytes = {0x61};
String string = new String(bytes, Charset.forName("UTF-8"));
System.out.println(string);

This will work, provided that the byte array really contains UTF-8 encoded stream of bytes. And it must be without BOM, otherwise the conversion will probably fail. Make sure that what you are trying to convert does not start with the sequence 0xEF 0xBB 0xBF.

Chinese Strings Handling in Java?

3 Answers3