How to fix missing double quotes issue when parsing JSON data?

Question

I am running a piece of code in Python3 where I am consuming JSON data from the source. I don't have control over the source. While reading the json data I am getting following error:

simplejson.errors.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2

Here is the code


import logging
import simplejson as json
logging.basicConfig(level=logging.INFO)


consumer = KafkaConsumer(
        bootstrap_servers='localhost:9092',        
        api_version=(1,0,0))

consumer.subscribe(['Test_Topic-1'])

for message in consumer:
    msg_str=message.value    
    y = json.loads(msg_str)
    print(y["city_name"])

As I can not change the source, I need to fix it at my end. I found out this post helpful as my data contains the timestamps with : in it: How to Fix JSON Key Values without double-quotes?

But it also fails for some values in my json data as those values contain : in it. e.g.

address:"1600:3050:rf02:hf64:h000:0000:345e:d321"

Is there any way where I can add double quotes to keys in my json data?

Could you provide a [Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example) please — Lucan, Jun 09 '20 at 20:38
- Edited the original question and added the code I am using. — sgmbd, Jun 09 '20 at 21:10
you can try with module `dirtyjson` - it can fix some mistakes. — furas, Jun 09 '20 at 21:19
You still haven't shown us an example of your input. In any case, it sounds like what you're dealing with _isn't JSON_. — ChrisGPT was on strike, Jun 09 '20 at 21:37

score 4 · Accepted Answer · answered Jun 09 '20 at 21:29

You can try to use module dirtyjson - it can fix some mistakes.

import dirtyjson

d = dirtyjson.loads('{address:"1600:3050:rf02:hf64:h000:0000:345e:d321"}')

print( d['address'] )

d = dirtyjson.loads('{abc:"1:2:3:4", efg:"5:6:7:8", "hij":"foo"}')

print( d['abc'] )

It creates AttributedDict so it may need dict() to create normal dictionary

d = dirtyjson.loads('{abc:"1:2:3:4", efg:"5:6:7:8", "hij":"foo"}')

print( d )

print( dict(d) )

Result:

AttributedDict([('abc', '1:2:3:4'), ('efg', '5:6:7:8'), ('hij', 'foo')])

{'abc': '1:2:3:4', 'efg': '5:6:7:8', 'hij': 'foo'}

score 1 · Answer 2 · answered Jun 09 '20 at 21:01

I think your problem is that you have strings like this:

{abc:"1:2:3:4", efg:"5:6:7:8", "hij":"foo"}

which are not valid JSON. You could try to repair it with a regular expression substitution:

import re
jtxt_bad ='{abc:"1:2:3:4", efg:"5:6:7:8", "hij":"foo", klm:"bar"\n}'
jtxt = re.sub(r'\b([a-zA-Z]+):("[^"]+"[,\n}])', r'"\1":\2', jtxt_bad)

print(f'Original: {jtxt_bad}\nRepaired: {jtxt}')

The output of this is:

Original: {abc:"1:2:3:4", efg:"5:6:7:8", "hij":"foo", klm:"bar"
}
Repaired: {"abc":"1:2:3:4", "efg":"5:6:7:8", "hij":"foo", "klm":"bar"
}

The regular expression \b([a-zA-Z]+):("[^"]+"[,\}]) means: boundary, followed by one or more letters, followed by a :, followed by double-quoted string, followed by one of ,, }, \n. However, this will fail if there is a quote inside the string, such as "1:\"2:3".

How to fix missing double quotes issue when parsing JSON data?

2 Answers2