9

I am getting the parameter value as parameter from the Jersey Web Service, which is in Japaneses characters.

Here, 'japaneseString' is the web service parameter containing the characters in japanese language.

   String name = new String(japaneseString.getBytes(), "UTF-8");

However, I am able to convert a few sting literals successfully, while some of them are creating problems.

The following were successfully converted:

 1) アップル
 2) 赤
 3) 世丕且且世两上与丑万丣丕且丗丕
 4) 世世丗丈

While these din't:

 1) ひほわれよう
 2) 存在する

When I further investigated, i found that these 2 strings are getting converted in to some JUNK characters.

 1) Input: ひほわれよう        Output : �?��?��?れよ�?�
 2) Input: 存在する            Output: 存在�?�る

Any idea why some of the japanese characters are not converted properly?

Thanks.

Janak
  • 4,986
  • 4
  • 27
  • 45
  • Why encoding and decoding a string in the same line? That does not make sense. Obviously, `japaneseString` is already a string, so you can simply use that in the code instead of `name`. As an answer: `String.getBytes()` uses the platform's default encoding for creating the byte array. And this might not be UTF-8. The decoding process with UTF-8 then will produce the junk chars. – Seelenvirtuose Jun 03 '14 at 07:18
  • [The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)](http://www.joelonsoftware.com/articles/Unicode.html) – Seelenvirtuose Jun 03 '14 at 07:27
  • Its a silly question but I have to ask it anyway: you're sure the font used can actually render all those characters, right? – Gimby Jun 03 '14 at 07:46
  • @Gimby: Yes, the others are rendering absolutely right. – Janak Jun 03 '14 at 08:10
  • The 3rd and 4th strings does not look like Japanese. I am not sure if UTF for Kanji and Hiragana are the same. – user2063329 Jun 25 '18 at 21:18

3 Answers3

13

You are mixing concepts here.

A String is just a sequence of characters (chars); a String in itself has no encoding at all. For what it's worth, replace characters in the above with carrier pigeons. Same thing. A carrier pigeon has no encoding. Neither does a char. (1)

What you are doing here:

new String(x.getBytes(), "UTF-8")

is a "poor man's encoding/decoding process". You will probably have noticed that there are two versions of .getBytes(): one where you pass a charset as an argument and the other where you don't.

If you don't, and that is what happens here, it means you will get the result of the encoding process using your default character set; and then you try and re-decode this byte sequence using UTF-8.

Don't do that. Just take in the string as it comes. If, however, you have trouble reading the original byte stream into a string, it means you use a Reader with the wrong charset. Fix that part.

For more information, read this link.

(1) the fact that, in fact, a char is a UTF-16 code unit is irrelevant to this discussion

fge
  • 119,121
  • 33
  • 254
  • 329
  • Please see the JAVA Doc it is not re-encoding the bytes again in UTF-8. http://docs.oracle.com/javase/6/docs/api/java/lang/String.html#String(byte[], java.nio.charset.Charset) Constructs a new String by decoding the specified array of bytes using the specified charset. – Nitul Jun 03 '14 at 07:27
  • @Nitul yes it does. See the documentation for `.getBytes()` and the constructor which is called here. – fge Jun 03 '14 at 07:28
  • Can you give the solution, how to decode the string into UTF-8, without calling getBytes method(). – Nitul Jun 03 '14 at 07:30
  • 3
    @Nitul read the answer again; a String has no notion of decoding or encoding at all; you only encode char sequences to byte sequences, and decode byte sequences into char sequences. The initial problem, if any, is that the OP fails to _decode_ the byte sequence properly. – fge Jun 03 '14 at 07:33
  • 2
    @Nitul UTF-8 is an encoding! You do not _decode a string into UTF-8_. A string is a simple char array, and chars do not have any inherent encoding. A character is simply a value pointing into a character table. An encoding is used for storing that character (the character's value) as one or more bytes. Nothing else. And the decoding process is the opposite: Take some bytes and create characters according to the used encoding. – Seelenvirtuose Jun 03 '14 at 07:35
  • 1
    Even if the `String` class most probably will be implemented using a sequence or array of `char`s, the definition of a `String` instance is even more abstract: 'The String class represents character strings.' The String class can just as well use an UTF-8 encoded byte array or an int array with code points as internal storage and still fulfill the API contract. – jarnbjo Jun 03 '14 at 07:56
4

I concur with @fge.

Clarification

In java String/char/Reader/Writer handle (Unicode) text, and can combine all scripts in the world.

And byte[]/InputStream/OutputStream are binary data, which need an indication of some encoding to be converted to String.

In your case japaneseStingr should already be a correct String, or be substituted by the original byte[].

Traps in Java

Encoding often is an optional parameter, which then defaults to the platform encoding. You fell in that trap too:

String s = "...";
byte[] b = s.getBytes(); // Platform encoding, non-portable.
byte[] b = s.getBytes("UTF-8"); // Explicit
byte[] b = s.getBytes(StandardCharsets.UTF_8); // Explicit,
                         //  better (for UTF-8, ISO-8859-1)

In general avoid the overloaded methods without encoding parameter, as they are for current-computer only data: non-portable. For completeness: classes FileReader/FileWriter should be avoided as they even provide no encoding parameters.

Error

japaneseString is already wrong. So you have to read that right. It could have been read erroneouslyas Windows-1252 (Windows Latin-1) and suffered when recoding to UTF-8. Evidently only some cases get messed up.

Maybe you had:

String japanesString = new String(bytes);

instead of:

String japanesString = new String(bytes, StandardCharsets.UTF_8);

At the end:

String name = japaneseString;

Show the code for reading japaneseString for further help.

Joop Eggen
  • 107,315
  • 7
  • 83
  • 138
4

Try with JVM parameter file.encoding to set with value UTF-8 in startup of Tomcat(JVM). E.x.: -Dfile.encoding=UTF-8

Nitul
  • 997
  • 12
  • 35