0

TL:DR: I'm trying to pass with curl a JSON content and getting failed.

I have a file with the following content:

test_api.txt content:

data={"crt":"t45","res_type":1,"stype":"standard","config":"{u'global': {u'lang': [], u'floor': u'1', u'tmax': 1000, u'dsp': [{u'status': u'disable', u'id': u'92'}], u'request_at': 3, u'crt_list': [{u'width': u'25', u'height': u'50', u'stype': u'restype', u'format': u'restype'}], u'request': True, u'geo': [{u'geo_id': u'21', u'floor': u'50'}]}}","sl_org":555,"parse_to":1111,"group_id":2020,"name":"test_api"}

The value of data is valid json, as far as I understand, because all online validators are agree that it's valid. And actually this is a result of python's json.dumps().

And I'm sending it this way:

curl -v -X POST 'https://example.com/api/v2/entity/create' -H "Content-Type: application/x-www-form-urlencoded" -d  "@test_api.txt"

What I see if i print POST conent:

<QueryDict: {u'data': [u'{"crt":"t45","res_type":1,"stype":"standard","config":"{u\'global\': {u\'lang\': [], u\'floor\': u\'1\', u\'tmax\': 1000, u\'dsp\': [{u\'status\': u\'disable\', u\'id\': u\'92\'}], u\'request_at\': 3, u\'crt_list\': [{u\'width\': u\'25\', u\'height\': u\'50\', u\'stype\': u\'restype\', u\'format\': u\'restype\'}], u\'request\': True, u\'geo\': [{u\'geo_id\': u\'21\', u\'floor\': u\'50\'}]}}","sl_org":555,"parse_to":1111,"group_id":2020,"name":"test_api"}']}>

Next I parse that code with json.loads() anf I get the following dict:

{u'stype': u'standard', u'config': u"{u'global': {u'lang': [], u'floor': u'1', u'tmax': 1000, u'dsp': [{u'status': u'disable', u'id': u'92'}], u'request_at': 3, u'crt_list': [{u'width': u'25', u'height': u'50', u'stype': u'restype', u'format': u'restype'}], u'request': True, u'geo': [{u'geo_id': u'21', u'floor': u'50'}]}}", u'name': u'test_api', u'parse_to': 1111, u'res_type': 1, u'crt': u't45', u'sl_org': 555, u'group_id': 2020}

And now I have to parse the value of 'config' key in this dict, that actually contents a string with another JSON-field:

value = u"{u'global': {u'lang': [], u'floor': u'1', u'tmax': 1000, u'dsp': [{u'status': u'disable', u'id': u'92'}], u'request_at': 3, u'crt_list': [{u'width': u'25', u'height': u'50', u'stype': u'restype', u'format': u'restype'}], u'request': True, u'geo': [{u'geo_id': u'21', u'floor': u'50'}]}}"

And here is the problem: any try to parse this string with json with json.loads(value) is going to be failed:

"ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"

Yes, it has single quotes inside of double quotes and it doesn't look right but I'm getting this string with default json.loads(), as you can see, so that what python does.

And my question is - what is going wrong and how to fix it? I'm passing valid json through standard symmetric python functions and getting broken json a result. How can I fix this pipeline and get this json parse correctly?

Paul
  • 6,641
  • 8
  • 41
  • 56

1 Answers1

1

Your outer structure is JSON, but the inner content is not JSON but a Python literal; thus, you need to use Python-syntax-aware tools, not JSON tools, for it.

import json, ast

json_str = '''{"crt":"t45","res_type":1,"stype":"standard","config":"{u'global': {u'lang': [], u'floor': u'1', u'tmax': 1000, u'dsp': [{u'status': u'disable', u'id': u'92'}], u'request_at': 3, u'crt_list': [{u'width': u'25', u'height': u'50', u'stype': u'restype', u'format': u'restype'}], u'request': True, u'geo': [{u'geo_id': u'21', u'floor': u'50'}]}}","sl_org":555,"parse_to":1111,"group_id":2020,"name":"test_api"}'''
data = json.loads(json_str)
config = ast.literal_eval(data['config'])

print(config['global']['tmax']) # emits 1000
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Hmm, I was pretty sure that inner 'json' is a result of json.dumps(value), it's an actually a form field was dumped. Let me check it, please. – Paul Aug 21 '19 at 16:29
  • It looks like you are right, it' just a string. Thanks a lot! – Paul Aug 21 '19 at 17:15