UTF-8 encoded characters from REST-query not rendered properly

Question

I'm consuming an external REST service that provides all content as UTF-8 encoded.

For some reason my application cannot properly handle the response. If I dump the response I will se things like LuleÃ¥ (should be Luleå).

EDIT: The same behavior happens if i forward (without altering) the string to the UI, ex.:

flash.message = "Test" + integrationService.testEncoding()

What I did was to create a _Events.groovy file in the /script folder and specifying there that

eventConfigureTomcat = { tomcat ->
    tomcat.connector.URIEncoding = "UTF-8"
    tomcat.connector.useBodyEncodingForURI = true
}

I also have the following in my Config.groovy:

grails.views.gsp.encoding = "UTF-8"
grails.converters.encoding = "UTF-8"

But that changed nothing. The response is still wrongly shown. I'm not sure if this is a configuration issue with Grails, with the embedded tomcat or with something else. I'm currently running my test setup on windows 7, but the same issue happens on my server running on Centos. Please advice.

EDIT2: If i consume the REST service using curl, everything is rendered correctly in the output.

EDIT3: I'm using org.springframework.web.client.RestTemplate and HttpComponents to consume the service:

private static final HttpHeaders requestHeaders
static{
    requestHeaders = new HttpHeaders()
    requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json")
    requestHeaders.set(HttpHeaders.ACCEPT, "application/json")
    requestHeaders.set("Accept-Encoding", "gzip")
}

private final static RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory(
    HttpClientBuilder.create().build()))
...
...
public def testEncoding(){
    ResponseEntity<String> response = restTemplate.exchange(
            "https://www.url.com", HttpMethod.GET, new HttpEntity<Object>(requestHeaders),
            String.class)
    def gamesJson = JSON.parse(response.getBody())
    //...
    //parse value from gamesJson
    //...
    return testValue
}

Are you sure that the place you "dump" the response is properly able to handle UTF-8? If you're just doing a `println` on the Windows console, for example, then you might be printing UTF-8 bytes but having the console interpret them as if they were a single byte encoding like windows-1252 — Ian Roberts, Feb 12 '15 at 15:19
@ian roberts Probably not. If I query the api with a browser (IE) I get a response that has the same character encoding problem. I I try to output the queried "text" in a gsp page, I have the same encoding problem. I'm a bit at a loss here about what is causing this. — john, Feb 12 '15 at 19:31
Can you create a sample app replicating the issue? or at least show the code where the service is consumed? Are you using rest client builder. Try setting the `content-type` in `headers` as `"application/json;charset=utf-8"` — dmahapatro, May 19 '15 at 19:13
@dmahapatro the http response has those in the header. Added example code. I can make an example app if neccessary. — john, May 19 '15 at 21:04
Now as I said earlier to use the content type as `application/json;charset=utf-8`. Modify as `requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json;charset=utf-8")` — dmahapatro, May 20 '15 at 16:50
@dmahapatro, so I receive UTF-8 bytes, but they are shown as ANSI by the browser. Yet the page contains ``. Now I'm really lost, why is the browser not rendering utf-8 bytes as utf-8? — john, May 20 '15 at 20:08
Refer http://stackoverflow.com/questions/7809931/how-to-force-utf-8-encoding-in-browser — dmahapatro, May 20 '15 at 20:17
@dmahapatro Thank you for your help, but I finally managed to find a solution [here](http://stackoverflow.com/questions/27603782/java-spring-resttemplate-character-encoding), mainly thanks to you leading me in the right direction. — john, May 21 '15 at 18:40

score 8 · Accepted Answer · edited May 23 '17 at 12:10

Per my previous answer:

You just need to add the StringHttpMessageConverter to the template's message converters:

RestTemplate template = new RestTemplate();
template.getMessageConverters()
        .add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
ResponseEntity<Object> response = template.exchange(endpoint, method, entity, 
                                                    Object.class);

score 3 · Answer 2 · answered May 20 '15 at 07:35

3

The encoding type can be enforced in the environment itself.

JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8 -Dclient.encoding.override=UTF-8

Just try setting the above encoding settings in windows/linux. I hope this should resolve the issue.

In this case, JVM will pickup the default encoding type from environment variable.

answered May 20 '15 at 07:35

J.P

495
1
6
15

This does not change the behavior. Problem persists. – john May 20 '15 at 16:36
i'm using groovy `RestConnector` on cygwin+windows and this solve a problem connecting to a service (elasticsearch), thanks! – genuinefafa Dec 15 '17 at 17:42

score 2 · Answer 3 · edited Jun 20 '20 at 09:12

Our team experienced a similar issue before, we have a 3rd party service and they said their output is encoded in UTF-8. But the strings returned are still garbled. After a bit of testing, it turns out that they were returning ISO-8859-1 encoded strings. What we did was to decode/encode their input into UTF-8 encoded characters so we can use those properly.

For your case, I think this is a similar issue:

UTF-8: Luleå

ISO-8859-1: LuleÃ¥

In Java, we did something like this:

Charset initialEncoding = Charsets.ISO_8859_1;
Charset outputEncoding = Charsets.UTF_8; 
byte[] byteArray = input.getBytes(initialEncoding);
String output = new String(new String(byteArray, outputEncoding));

In Groovy, I think you could do something like

import groovy.json.JsonSlurper;

def main = {
    def response = '{"name":"LuleÃ¥"}' 

    def slurper = new JsonSlurper()
    def result = slurper.parse(response.getBytes(), 'UTF-8')

    println result.name // prints Luleå
}

Thank you for the effort! However, the solution was much [easier](http://stackoverflow.com/questions/27603782/java-spring-resttemplate-character-encoding). — john, May 21 '15 at 18:42

score 2 · Answer 4 · edited May 23 '17 at 10:29

2

The answer to my problem is already found on Stack Exchange.

You just need to add the StringHttpMessageConverter to the template's message converters:

restTemplate.getMessageConverters()
    .add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));

edited May 23 '17 at 10:29

Community

1
1

answered May 22 '15 at 06:12

john

613
1
7
25

score 0 · Answer 5 · answered Mar 16 '20 at 08:47

In my case that i had the same problem with contents received from my REST web service from the server and not my local enviroment. I had search a lot and finally i found a solution that resolved my issue. In Windows I added a new environment variable with key: JAVA_TOOL_OPTIONS and set its value to: -Dfile.encoding=UTF8. The (Java) System property will be set automatically every time a JVM is started. You will know that the parameter has been picked up because the following message will be posted to System.err:

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8

UTF-8 encoded characters from REST-query not rendered properly

5 Answers5

Linked