How can I deserialize JSON with u' unicode markup in Java?

Question

I have this HTTP response body to de-serialize:

String response = "result : {'url': u'https://somedomain/', 'fields': {'policy': 
u'eyJjb25kaXRpb25zIjogW1siYfgfhudGVudC1sZjMyWiJ9', 'AWSAccessKeyId': 
u'ASIccccccNA', 'x-amz-security-token': 'FQofgF', 'key': u'bbb.file', 
'signature': u'rm9gdflkjfs='}}"

I am using the jackson.core (2.9.0) java package and lib (have also tried GSON) and get this error:

Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: 
Unrecognized token 'u': was expecting ('true', 'false' or 'null')

Deserialization code:

MyResponse deserializedResponse = new ObjectMapper()
  .configure(JsonParser.Feature.ALLOW_SINGLE_QUOTES, true)
  .configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true)
  .configure(JsonGenerator.Feature.ESCAPE_NON_ASCII, true)
                        .readValue(response, MyResponse.class);

I have considered something like this but it feels like there should be a better / safer way:

String sanitizedResponse = response.replaceAll("u'", "'");

--

Using Java 1.8.

Any help appreciated.

From where do you get the response body? Is it somehow related to python? Because I found an answer which might help you here: https://stackoverflow.com/a/34479722/2507819. Nevertheless I think the solution is to get the json in the format you need. The additional "u" just stand for unicode — JanTheGun, Nov 29 '17 at 09:56
No probs. It is actually from a aws lambda (python) so you are right there. Have tried modifying the response but it seems I can't do that. — Willeman, Nov 29 '17 at 10:07
Your approach can potentially distort your data. Consider the result of this: `String response = "u'my output with u'"` It will replace the last 'u' as well. You will need to write a regex that just considers the first 'u'' — kosgeinsky, Nov 29 '17 at 12:15
@kosgeinsky yup exactly. And writing a fail-safe regex isn't trivial. That is the point of this question. — Willeman, Nov 29 '17 at 12:21

JanTheGun · Accepted Answer · 2017-11-29T15:01:40.890

As python caused this problem I think the best solution is to let python fix it ;-). Fortunately with jython you can stick with a pure java implementation.

First you need to add the jython standalone dependency in your pom.xml:

<dependencies>
    <dependency>
        <groupId>org.python</groupId>
        <artifactId>jython-standalone</artifactId>
        <version>2.7.1</version>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.5</version>
    </dependency>
</dependencies>

(As you can see I also used apache commons io for my example so I added it as well)

I put your (invalid) json string into the text file "c:/temp/json.txt" which has the following content:

{'url': u'https://somedomain/', 'fields': {'policy': 
u'eyJjb25kaXRpb25zIjogW1siYfgfhudGVudC1sZjMyWiJ9', 'AWSAccessKeyId': 
u'ASIccccccNA', 'x-amz-security-token': 'FQofgF', 'key': u'bbb.file', 
'signature': u'rm9gdflkjfs='}}

Now here is the code to read the json file, set up the Python Interpreter and handover the json to clean it up:

String content = FileUtils.readFileToString(new File("c:/temp/json.txt"), "UTF-8");

PythonInterpreter pi = new PythonInterpreter();
pi.exec("import json");
pi.exec("jsondata = " + content);
pi.exec("jsonCleaned = json.dumps(jsondata)");
PyObject jsonCleaned = (PyObject) pi.get("jsonCleaned");
System.out.println(jsonCleaned.asString());
pi.close();

The output is:

{"url": "https://somedomain/", "fields": {"signature": "rm9gdflkjfs=", "AWSAccessKeyId": "ASIccccccNA", "x-amz-security-token": "FQofgF", "key": "bbb.file", "policy": "eyJjb25kaXRpb25zIjogW1siYfgfhudGVudC1sZjMyWiJ9"}}

When you put that in a json validator (https://jsonlint.com/) you can see that it is a valid json now. I can't tell if the performance is good enough for your use case so you have to test that out.

Remark: In Eclipse there seems to be a bug with that jython version. It shows the following error:

console: Failed to install '': java.nio.charset.UnsupportedCharsetException: cp0.

Although it works nevertheless you can get rid of it by adding the following VM-Argument to your Run-Configuration:

-Dpython.console.encoding=UTF-8

Remark2: For the sake of completeness and to fully answer that question - here is how you can deserialize the cleaned JSON:

Add GSON Dependency to your pom.xml:

<dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.8.2</version>
</dependency>

Create representing classes:

Info class

public class Info {
    private String url;
    private Fields fields;

    public String getUrl() {
        return url;
    }

    public void setUrl(String url) {
        this.url = url;
    }

    public Fields getFields() {
        return fields;
    }

    public void setFields(Fields fields) {
        this.fields = fields;
    }

}

Fields class

import com.google.gson.annotations.SerializedName;

public class Fields {
    private String signature;
    private String AWSAccessKeyId;

    @SerializedName("x-amz-security-token")
    private String x_amz_security_token;

    private String key;
    private String policy;

    public String getSignature() {
        return signature;
    }

    public void setSignature(String signature) {
        this.signature = signature;
    }

    public String getAWSAccessKeyId() {
        return AWSAccessKeyId;
    }

    public void setAWSAccessKeyId(String aWSAccessKeyId) {
        AWSAccessKeyId = aWSAccessKeyId;
    }

    public String getX_amz_security_token() {
        return x_amz_security_token;
    }

    public void setX_amz_security_token(String x_amz_security_token) {
        this.x_amz_security_token = x_amz_security_token;
    }

    public String getKey() {
        return key;
    }

    public void setKey(String key) {
        this.key = key;
    }

    public String getPolicy() {
        return policy;
    }

    public void setPolicy(String policy) {
        this.policy = policy;
    }

}

Finally add the following code after you get your cleaned JSON:

Gson gson = new Gson();
Info info = gson.fromJson(jsonCleaned.asString(), Info.class);

Thanks a lot for the detailed answer. Will definitely check it out asap and accept. — Willeman, Nov 29 '17 at 15:19

kosgeinsky · Answer 2 · 2017-11-29T13:37:32.390

0

You need to use the following regex that replaces u' from beginning of word boundaries to replace with '

String regexPattern = "(\\bu')";

edited Nov 29 '17 at 13:37

answered Nov 29 '17 at 12:36

kosgeinsky

508
3
21

Have fun with input like `{' u':123}`. – Joey Nov 29 '17 at 12:38

How can I deserialize JSON with u' unicode markup in Java?

2 Answers2