Japanese character not showing properly converting CSV file

Question

I am converting CSV file from Tatoeba project. It contains Japanese characters. I am inserting data into SQLite database. Insertion is going without a problem, but characters are showing not properly. If I insert directly:

            String str = content_parts[2];
            sentence.setValue(str);

Getting values like this:

ãã¿ã«ã¡ãã£ã¨ãããã®ããã£ã¦ãããã

I have tried to decode to UTF8 from JIS:

            String str = content_parts[2];
            byte[] utf8EncodedBytes = str.getBytes("JIS");
            String s = new String(utf8EncodedBytes, "UTF-8");
            sentence.setValue(s);

JIS:

$B!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!r!)!)!/!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)!r!)!)!)!)!)!)!)!)!)!)!)!)!)!)!)(B

Shift-JIS:

????\??????�N?�}??????????????????��?????�N?�N???��??????

Shift_JIS:

????\????????????????????????��?�N??????????????????��??????

CSV file (when opened by Excel 2010)

n гЃЌгЃїгЃ«гЃЎг‚‡гЃЈгЃЁгЃ—гЃџг‚‚гЃ®г‚’г‚‚гЃЈгЃ¦гЃЌгЃџг‚€гЂ‚

What I am doing wrong? How to solve this problem?

May be the issue is with the database encoding, not with java display — ortis, Feb 12 '15 at 08:54
@immibis, honestly, i do not know. now i am searching for how to identify encoding — Joe Rakhimov, Feb 12 '15 at 08:56
@ortis, I have tried to System.out.println(s). Database and values in console are same — Joe Rakhimov, Feb 12 '15 at 08:57
Well, if you cant display correct word within DB client, it is probably a databse encoding issue. Check if the encoding is UTF8 — ortis, Feb 12 '15 at 08:58

score 2 · Answer 1 · edited May 23 '17 at 12:17

If you are still searching for solution, refer below link

setting-a-utf-8-in-java-and-csv-file and handle Japanese characters

csv-reports-not-displaying-japanese-characters

In brief, add BOM(byte order mark) characters to your file outputstream before passing it to outputstream writer.

String content="some string to write in file(in any language)";

FileOutputStream fos = new FileOutputStream("D:\csvFile.csv");

fos.write(239);

fos.write(187);

fos.write(191);

Writer w = new BufferedWriter(new OutputStreamWriter(fos, StandardCharsets.UTF_8));

w.write(content);

w.close();

Hope this will help

Japanese character not showing properly converting CSV file

1 Answers1