1

I am having a csv file which has some fields having chinese character strings. Unfortunately i dont know what is encoding of this input csv file. I am trying to read this input csv and using selective fields from it, i am making a html and another csv file as output.

While reading csv input, i tried all encoding from list http://docs.oracle.com/javase/7/docs/technotes/guides/intl/encoding.doc.html which have Chinese mentioned in their description. And found if I use 

InputStreamReader read = new InputStreamReader(filepath,"GB18030");

for reading csv and 

OutputStreamWriter osW=new OutputStreamWriter(objBufferedOutputStream,"UTF-16");

For writing html and csv, my output doesnt show weird characters.

But, there are 2 problems:

  1. The output is showing strings which are altogether different from input ! I mean, even when im not doing any processing on any string from my code, the output is not found in any field of input csv.

For example, my input has a chinese char string: 陈真珍 on field number 8. but my output html has something like: 闄堢湡鐝� which corresponds to input field number 8.

  1. as u can see, there is a questionmark, i.e. replacement char from unicode in output 闄堢湡鐝�

I request you to kindly help me trace where can be a mistake here...

PS: Aiso, I checked Google translation and found,input string 陈真珍 means some Chen Zhen Zhen

and its corresponding output string 闄堢湡鐝� means something called as Yaobaoyujue So there is difference in meaning as well as representation of characters also.

Afshin Moazami
  • 2,092
  • 5
  • 33
  • 55
adityag
  • 603
  • 3
  • 8
  • 24
  • 1
    Could you please use `file` command on Linux/UNIX/GNU-win32 to check your CSV encoding? And have you ever tried UTF-8 and UTF-16? – longhua Oct 29 '13 at 09:38
  • Thanks ! I tried file command and it showed UTF-8 encoding for input csv so now I am using UTF-8 for reading and UTF-16 for writing html and now i can see exactly same input and output strings and there is no replacement character ! – adityag Oct 29 '13 at 09:54

2 Answers2

2

That output means that your input is NOT in GB18030 encoding.

Also: please check and double-check how you view your files: what encoding does the program use that opens the files, specifically the input file. Usually text files (and CSV files) don't come with metadata attached to them that shows their encoding, so the editors have to guess and that guess can easily be wrong.

Joachim Sauer
  • 302,674
  • 57
  • 556
  • 614
0

Please keep the enconding be consistent when reading / writing Chinese character. Since some Chinese character may not be represented by the all the encodings, such as GBK, GB18030 etc.

You can have a try to use UTF-8 enconding to handle Chinese character.

Mengjun
  • 3,159
  • 1
  • 15
  • 21