0

I am encoding a 40kb dictionary of dictionaries and lists into json and then pushing it over http to a nosql database. I've used both jsonpickle.encode and json.dumps modules to encode my dictionary's content, but both are leading to an http error. I tried manually CURLing the problematic section of code with a result of the error "-bash: syntax error near unexpected token `('"

Here is some example code:

import urllib2 , jsonpickle
url = "http://amazonaws.com/server/%s/_create" % item
data = jsonpickle.encode( some_dict_of_dicts_and_lists ) # also tried json.dumps here.
try:
    req = urllib2.Request ( url , data , { 'Content-Type' : 'application/json' } )
    f = urllib2.urlopen ( req )
except Exception as e:
    print "Error: %s" %e

The above works for entering certain sections of my dictionary into my nosql db. However, this prints "Error: HTTP Error 400: Bad Request" when I send over other sections of my dictionary. This means to me that something is not getting encoded properly in the data variable/string. To get CURL's response on this problem, I tried CURLing the following code:

item_id = item_dictionary [ 'id' ]
data = jsonpickle.encode( some_dict_of_dicts_and_lists ) # also tried json.dumps here.

command = 'curl -XPOST "http://amazonaws.com/server/%s/_create" -d '"%s"' % ( nsn_id , data )
os.system(command)

This produces the error "sh: -c: line 0: syntax error near unexpected token `('"

If I try to manually type the curl command into the command line, I get the following set of errors:
curl: (6) Could not resolve host: material; nodename nor servname provided, or not known
curl: (6) Could not resolve host: items; nodename nor servname provided, or not known
curl: (6) Could not resolve host: of; nodename nor servname provided, or not known
curl: (3) [globbing] unmatched close brace/bracket at pos 146
{"error":"MapperParsingException[Failed to parse]; nested: JsonParseException[Unexpected end-of-input in VALUE_STRING\n at [Source: [B@2ff246ab; line: 1, column: 6519]]; ","status":400}

So:
1) Is there a better way to make sure the the json encode process captures makes all of the parentheses and single quotes are formatted with a '\' in front of them? Should I do a replace on all of them?

2) Is there a good way to figure out the cause of this error in a more detailed way using urllib2 or do I need to do CURL from the command line in order to get the bash type error?

ccdpowell
  • 629
  • 5
  • 14
  • 22
  • You should try just using the built-in `import json; json.dumps` for this, like [here](http://stackoverflow.com/questions/3290522/urllib2-and-json). But why do you assume that the problem is the formatting of the json dictionary? And what precisely did you try CURLing? – David Robinson Jan 24 '13 at 08:46

1 Answers1

0

To answer both your questions:

1) json.dumps - assuming that you pass it valid input - will always create a string representing a valid JSON object. If you pass invalid input, it will throw an exception. Python's json module is extremely widely used and if there were a bug that could cause it to spit out an invalidly formatted JSON dictionary then this would definitely be known (and fixed) by now.

2) You're getting the HTTPError because the web server is returning a page with a HTTP Status Code of 400 in response to your request. There was probably also some content in the page body, though, and urllib2 hasn't thrown that away. You can call the .read() method of the HTTPError to get any content that was on the page, rather than just seeing the status code like you are at the moment (which is sometimes not enough to understand what you've done wrong).

Most web APIs will return you some sort of useful explanation when spitting out a 400 - Bad Request error, such as a message telling you what parameter you missed out in your request. Putting print e.read() in your exception block will let you see this, and will probably allow you to solve your current problem.

Mark Amery
  • 143,130
  • 81
  • 406
  • 459