46

I am building a mobile app with Flutter.

I need to fetch a json file from server which includes Japanese text. A part of the returned json is:

{
     "id": "egsPu39L5bLhx3m21t1n",  
     "userId": "MCetEAeZviyYn5IMYjnp",  
     "userName": "巽 裕亮",  
     "content": "フルマラソン完走に対して2018/05/06のふりかえりを行いました!"
}

Trying the same request on postman or chrome gives the expected result (Japanese characters are rendered properly in the output).

But when the data is fetched with Dart by the following code snippet:

  import 'dart:convert';
  import 'package:http/http.dart' as http;

  //irrelevant parts have been omitted    
  final response = await http.get('SOME URL',headers: {'Content-Type': 'application/json'});
  final List<dynamic> responseJson = json.decode(response.body)
  print(responseJson);

The result of the print statement in logcat is

{
     id: egsPu39L5bLhx3m21t1n, 
     userId: MCetEAeZviyYn5IMYjnp, 
     userName: å·½ è£äº®, 
     content: ãã«ãã©ã½ã³å®èµ°ã«å¯¾ãã¦2018/05/06ã®ãµãããããè¡ãã¾ããï¼
}

Note that only the Japanese characters (value of the content key) is turns into gibberish, the other non-Japanese values are still displayed properly.

Two notices are:

  1. If I try to display this Japanese text in my app via Text(), the same gibberish is rendered, so it is not a fault of Android Studio's logcat.
  2. If I use Text('put some Japanese text here directly') (ex: Text('睡眠')), Flutter displays it correctly, so it is not the Text widget that messes up the Japanese characters.
Tran Triet
  • 1,257
  • 2
  • 16
  • 34

2 Answers2

120

If you look in postman, you will probably see that the Content-Type http header sent by the server is missing the encoding tag. This causes the Dart http client to decode the body as Latin-1 instead of utf-8. There's a simple workaround:

http.Response response = await http.get('SOME URL',headers: {'Content-Type': 'application/json'});
List<dynamic> responseJson = json.decode(utf8.decode(response.bodyBytes));
Richard Heap
  • 48,344
  • 9
  • 130
  • 112
  • Thanks for the extra info regarding the encoding tag. That was helpful! Just to make sure I understand you correctly, when I use postman, there are 9 response headers, 2 of which are `Content-Encoding →gzip` and `content-type →application/json`. So I guess the `encoding` tag is not missing, but its value is wrong (e.g. the value should have been 'utf-8` instead of 'gzip'). Is that right? – Tran Triet Jul 17 '18 at 03:53
  • 8
    Actually, just the Content-Type header. With `Content-Type: application/json` Dart will assume Latin-1 encoding. With `Content-Type: application/json; charset=utf-8` Dart will use the specified charset to decode from bytes (response.bodyBytes) to characters (response.body). The work around is to decode the bytes yourself, since you know that the charset is utf-8. – Richard Heap Jul 17 '18 at 11:02
  • @RichardHeap, may I ask you to have a look at a Flutter related question here : https://stackoverflow.com/questions/60565658/fluter-image-picker-package-show-images-one-after-another-with-delete-action ? – Istiaque Ahmed Mar 26 '20 at 21:02
  • Isn't this behavior (technically) wrong according to https://www.ietf.org/rfc/rfc4627.txt? It states in its section 3: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8." So if the Content-Type is "application/json" (without the charset=utf-8), shouldn't http AUTOMATICALLY decode the body using utf-8? – pablete Sep 06 '21 at 23:10
  • @pablete that's just the default - it could be UTF-16BE, etc. Also, you wouldn't want `http` to implement special cases for particular content types. – Richard Heap Sep 08 '21 at 02:20
  • @RichardHeap I *would* like it for http to implement the correct enconding if is going to apply any transformation at all. For 'application/json' there IS NO ambiguity in the correct enconding so it should use it. Also look at this issue (with a lot of discussion both ways): https://github.com/dart-lang/http/issues/175 – pablete Sep 08 '21 at 16:20
7

So simple! Instead of using response.body; You should use utf8.decode(response.bodyBytes)

Hossein Azem
  • 331
  • 3
  • 8