Working with invalid JSON

Question

I'm working on a project that involves taking massive amounts of data-points from a server, in the form of JSON. For example, a response might include an array with 1000+ length.

To help compress some of this data, it's been suggested to ignore holes in the available data where no information is available. This project deals with changes in stock prices, and a stock isn't necessarily traded 7 days a week. The holidays and weekends are currently represented as just a comma, like so:

And so forth.

Is it possible to accept the JSON data, even if it's invalid? And if so, can I search for these non-existent values and give them an empty value just to account for them and make the JSON valid?

Thanks!

EDIT:: More information that might be relevant.

This data may be coming cross-domain as JSONP. What we're building is essentially an embeddable widget to go on some of our clients' webpages - the data is coming across as JSONP.

This might be a bit naive (and inception-y) - but could I pass this invalid JSON as a text value of a valid JSON field, and then fix it/parse it client-side?

Use http://jsonlint.com to test your JSON. But if it's not valid, you can't parse it as JSON, period. — Blazemonger, Apr 30 '13 at 13:54

Denys Séguret · Answer 1 · 2013-04-30T14:00:23.457

2

No, you can't just pass holes in JSON. An array can't have consecutive commas :

See json.org : enter image description here

What you can do is :

use a specific, predefined value (a string, a special number that you can't possibly receive normally) and replace it after parsing with undefined
use eval instead of JSON.parse (which is only a valid solution as you're also the data producer)

Personally, in this context, I use NaN (which is the industry standard to point missing numbers in number arrays) combined with eval (some libraries like gson enable the writing of NaN in JSON, as its absence is obviously unfortunate).

edited Apr 30 '13 at 14:00

answered Apr 30 '13 at 13:55

Denys Séguret

372,613
87
782
758

[`eval` is for quitters.](http://stackoverflow.com/questions/86513/why-is-using-the-javascript-eval-function-a-bad-idea) – Blazemonger Apr 30 '13 at 13:55
@Blazemonger Read the answers of the questions you link to. It's totally not pertinent in this context. – Denys Séguret Apr 30 '13 at 13:59

Matt · Accepted Answer · 2013-04-30T14:10:45.973

2

jQuery tries to detect the response format of a response and tries to validate it for you. Therefore, if it detects it as JSON (either through the Content-Type header of application/json, or via you setting the dataType option to json) , it will throw a parse error and reject it.

You can leave jQuery out of it and handle the parsing yourself by setting the dataType to text;

jQuery.ajax('/foo.html', {
    dataType: 'text'
}).done(jQuery.noop);

... then in your response handler, perform replacement operations on the string to turn your invalid JSON into valid JSON, then parse it using jQuery.parseJSON;

jQuery.ajax('/foo.html', {
    dataType: 'text'
}).done(function (data) {
    var formatted = data.replace(/,\s+,/g, ',null,');
    var parsed;

    try {
        parsed = jQuery.parseJSON(formatted);
    } catch (e) {
        // Handle errors;
        return;
    }

    // use parsed
});

If you are in control of the sender, you can also look at a solution such as hpack.

edited Apr 30 '13 at 14:10

answered Apr 30 '13 at 14:00

Matt

74,352
26
153
180

What if it's a cross-domain request? This is for an embeddable 'widget' that clients can place on their sites, so it makes a request back to us for the data. I know typically this needs to be a JSONP request, and I've expanded my original question to reflect this. – Joshua Longanecker Apr 30 '13 at 19:50
1

@JoshuaLonganecker: What you've got is valid JavaScript, albeit not valid JSON. JSONP is a bit of a misnomer; JSONP responses have to be valid JavaScript, not valid JSON. It should just *work* out of the box. Your callback function will receive an array as the parameter, and the elements removed will have the value `undefined`. – Matt Apr 30 '13 at 19:55
That's encouraging! We're not quite to the point of RESTful yet, but I'm hoping it will be that simple. I'll have to make it valid JSON before I can work with it, but if I can at least capture the information then I'll good. – Joshua Longanecker Apr 30 '13 at 20:11

score 0 · Answer 3 · answered Apr 30 '13 at 13:55

If it's always just a line with a comma on its own, then you could do something like a regex replace of \b,\b (or something rather smarter because this has some pitfalls) with 0, to insert 0's where there are 'empty' invalid lines and then parse the end result. The indentation doesn't matter so that should just work.

If the incoming data is broken but in a consistent manner, then there's nothing inherently wrong with simply trying to 'fix' it before parsing it as standard JSON.

score 0 · Answer 4 · answered Apr 30 '13 at 14:06

No. That's a bad suggestion. Suggest back to use gzip or another compression mechanism instead of breaking the format. This can lead others to want to break the format even more. At that point it's not JSON but your own format.

If you have to, send a nonsense value that you can detect, like -1 or "" instead of an empty array slot. These will shrink the file somewhat and still be valid.

Working with invalid JSON

4 Answers4