0

I have installed API Manager and APIM Analytics on CentOS 7.6, the version is 2.6.0. I find that Chinese characters are not displayed properly in the pages of the Analyze dropdown menu.

enter image description here

But the Chinese characters are able to be displayed properly in other pages

enter image description here

So this problem is focused on "APIM Analytics". I have checked the data in H2 database, I find that the data of H2 is correct, but the returned Json data is not correct.

enter image description here

and

enter image description here

I also checked a Shell file /usr/lib64/wso2/wso2am/wso2am-analytics-2.6.0/wso2/worker/bin/carbon.sh, I found that -Dfile.encoding=UTF8 has been configured.

enter image description here

I doubt whether the data is not encoded in the transmission process. Has anyone met this problem?

--- Additional content (2020/3/25) ---

I just checked deployment.yaml, which is at /usr/lib64/wso2/wso2am/wso2am-analytics-2.6.0/conf/worker. I did not find out any configuration item of character set.

enter image description here

I also reviewed usage.jag, which is at /usr/lib64/wso2/wso2am/2.6.0/repository/deployment/server/jaggeryapps/store/modules/statistics. As you can see, there is not any program logic of data query. I guess I'll have to read the source codes of wso2am-analytics-2.6.0. I don't think it is an efficient way.

enter image description here

--- Additional content (2020/3/26) ---

I found that some strange symbols are appended to the returned Json data(shown in the screenshot below). The returned Json data is as follows: å\u0085¨ç\u0090\u0083é£\u008eå\u0090\u0091æ\u009f¥è¯¢ (admin), normal Json data should look like this: \u6f22\u5b57.

enter image description here

Freeman
  • 1
  • 3
  • Try reading data as bytes. See if db-driver has any character encoding properties that you can set. – Ravindra HV Mar 24 '20 at 19:56
  • Hi Ravindra, thank you for your reply. I've looked some jaggery script files up in the deployment path, I found that the main program logic has been encapsulated. Probably I'll have to download and read the source code of wso2am-analytics-2.6.0, it means this problem can't be solved in a short term. :( BTW, I live in China. Because of the time difference, I may not be able to reply in time. Please forgive me. – Freeman Mar 25 '20 at 01:05
  • Please refer to " Additional content (2020/3/25)" in question. – Freeman Mar 25 '20 at 02:35
  • I am in India. I am not familiar with the framework. If modifying data is an option and is feasible, you could encode data in base64 in database and store and upon receiving it, decode it [its easy to perform base64 encode and decode in javascript - just use atob() and btoa()]. – Ravindra HV Mar 25 '20 at 12:24
  • PS: A better option would be to create a view that returns the data in hexadecimal format and you can convert it back to unicode using some function in javascript as mentioned here https://stackoverflow.com/questions/21647928/javascript-unicode-string-to-hex – Ravindra HV Mar 25 '20 at 12:38
  • Hi Ravindra, I've tested and verified the method that you mentioned, it works well. Now the problem is that some strange symbols are appended to the returned Json data(Please look at the screenshot that I have attached in question today). In my opinion, JavaScript can't handle these strange symbols. On the other side, I'm not familiar with the structure of the source code, I'll need some time to investigate them. Even if I could identify the problematic code , perhaps I would not be able to recompile and repackage these source code accurately, as we know, WSO2am is a complex project. – Freeman Mar 26 '20 at 02:48
  • Not sure. May be the framework is not handling unicode characters (or may be it does not handle the given set of unicode characters well - since there are variations w.r.t unicode). Did you try fetching the contents from db as base24 or hexadecimal encoded through a view? – Ravindra HV Mar 27 '20 at 00:43
  • Also when you say you tested and verified method then what are you referring to `atob()` / `bota()` or the hexadecimal approach in the link? – Ravindra HV Mar 27 '20 at 00:46
  • Good morning, Ravindra. I haven't tried it yet, I think that the data of WSO2AM_STATS_DB are normal. I'm still studying the source code of Analytics-apim project, I still suspect that the program does not deal with character set correctly. – Freeman Mar 27 '20 at 01:09
  • I just tested hexEncode() and hexDecode() in JavaScript, the hexadecimal approach in the link. – Freeman Mar 27 '20 at 01:27

1 Answers1

0

I find a suspicious point in the source code, the details are as follows:

Github repository is "wso2 / carbon-apimgt", Java file is "org.wso2.carbon.apimgt.impl.utils.APIUtil", the method is "executeQueryOnStreamProcessor".

The key code is “String responseStr = EntityUtils.toString(entity);”. According to document of HttpClient, EntityUtils.toString() method is going to use the default character set, which is ISO-8859-1, if other character set (For instance: UTF-8) is not specified for it.

So the point is that ISO-8859-1 can not process Chinese information properly.

I hope the developers of WSO2 can notice that ASAP.

Freeman
  • 1
  • 3