JSON Validation - Invalid character found

Question

When I hit the API its response has a special/weird character that is causing the JSON response to fail validation. Even when I try to escape the character JSON validator throw an error Escaped unescaped character. Now I'm trying to figure out if the problem comes from the backend or a Postman problem. Has anyone come across this problem where Postman return your JSON output with an invalid character that cannot be escaped?

{
    "page": 0,
    "totalRecords": 1,
    "totalPages": 1,
    "offset": 0,
    "status": "success",
    "entityList": [
        {
            "subjectcode": "HS   222",
            "coursenumber": "HS   222",
            "name": "Advanced First Aid",
            "departments": "HRHP",
            "description": "Advanced First Aid",
            "status": "A",
            "attributes": "",
            "lastoffered": "",
            "schedcode": "",
            "sectionstatus": "",
            "mincredithrs": "3.0",
            "maxcredithrs": "3.0",
            "coursenotes":"First Aid is the immediate care given to a person who has been injured or who suddenly becomes ill. It includes self-help and home care if more advanced medical assistance is not needed or is delayed. In Advanced First Aid, the student is awarded one credit for having displayed competence in both CPR and First Aid.&nbsp; The only certifications recognized are those of the American Heart Association.&nbsp; These courses do NOT need to be taken on this campus.&nbsp; However, they are offered on-line and through the University Ticket Office (http:/byui.universitytickets.com).&nbsp; For the on-line course, go to <a href=\"http://www.onlineaha.org/index.cfm?fuseaction=main.courseCatalog\">http://www.onlineaha.org/index.cfm?fuseaction=main.courseCatalog</a> and click on <a name=\"90-1401\">Heartsaver<sup></sup> First Aid CPR AED Online Part 1</a>&nbsp;under &quot;Workplace Training.&quot;&nbsp; Skills tests will need to be arranged after completing on-line portion of the course.&nbsp; To successfully complete the course, each qualifying student will show his/her curent American Heart Association's First Aid and CPR Certification cards.",
            "id": "HS   222 UG12"
        }
    ]
}

The invalid character is found in "coursenotes" value:

This thread should help https://stackoverflow.com/questions/6234773/can-i-escape-html-special-chars-in-javascript — Dipansh Khandelwal, Nov 29 '21 at 16:58
Replace the SUB character by `\u001a`. (Remember that the `\ ` has to exist in the JSON string at the end so if you are testing with a string literal, escape the backslash with another backslash!) — CherryDT, Nov 29 '21 at 17:18
...but I think this is just a symptom. If I google your text, I can see that it _should_ be `®` (registered trademark). I'm guessing that at some point between the database and the JSON validator (could even be inside the validator!) a character encoding problem arises and the `®` cannot be represented in the given encoding, causing it to be replaced with a SUB character (substitute character) - causing issues with the JSON string because JSON cannot contain most control characters including the SUB character. — CherryDT, Nov 29 '21 at 17:24
In regards to "cannot be escaped" - note that "escaping a character" for JSON doesn't mean prepending a `\ ` and calling it a day. That happens to work for `"` and `\ ` itself, but everything else needs a different representation, such as a newline that becomes `\n`. Any character, including those that don't have their own standard escape sequence like `\n`, can be escaped using `\uXXXX` with `XXXX` being the Unicode code point, so this character (which is the character U+001A) can be escaped using `\u001A`. — CherryDT, Nov 29 '21 at 17:35
@CherryDT, Thank you so much for your help. This totally helps me understand this a lot more. I'd also like to find out where the character encoding problem arises so I don't have to replace the SUB character manually every time I come across something like this. Could this be a Postman issue? — JCastillo, Nov 29 '21 at 17:50
I don't think so, but it's easy to check - use a different client or look at the raw HTTP request and response in Fiddler. But in that regard, maybe you have a restricted `Accept-Charset` header? You shoulnd't have any and servers shouldn't pay attention to them anymore, but who knows... — CherryDT, Nov 29 '21 at 18:01
@CherryDT When I check the response using a different client I get `\u001a` and even in Postman when I check its response in Raw view I get `\u001a`. The encoding problem arises in Pretty view. And in my headers I have `Accept-Encoding: gzip, deflate, br` — JCastillo, Nov 29 '21 at 18:27
Apparently it is a Postman bug that was supposedly fixed in version 7.3.6 and PostmanCanary. However the issue still persist in postman version 9.2.0 and in postmanCanary as well. Just wanted to share my finding — JCastillo, Nov 29 '21 at 20:54
Oh okay, I didn't know you copied from pretty. Since you were working with the data programmatically (or planning to do so, hence the validation) and not looking at it, I thought you would have copied the raw response data... — CherryDT, Nov 30 '21 at 07:16

0stone0 · Answer 1 · 2021-11-29T17:07:24.367

If we copy paste your json into a hidden-chars-shower like this one we can identify the weird char as an:

Substitute character

(U+001A)

In computer data, a substitute character (␚) is a control character that is used to pad transmitted data in order to send it in blocks of fixed size, or to stand in place of a character that is recognized to be invalid, erroneous or unrepresentable on a given device

Consider replacing it with a 'normal' char before parsing as JSON

Some related posts:

JSON Validation - Invalid character found

1 Answers1

Substitute character