5

I'm trying to convert my python script from issuing a curl command via os.system() to using requests. I thought I'd use pycurl, but this question convinced me otherwise. The problem is I'm getting an error returned from the server that I can see when using r.text (from this answer) but I need more information. Is there a better way to debug what's happening?

for what it's worth I think the issue revoles around converting my --data flag from curl/pycurl to requests. I've created a dictionary of the params i was passing to --data before. My guess is that one of those isn't valid but how can I get more info to know for sure?

example:

headers2 = {"Accept":"*/*", \
"Content-Type":"application/x-www-form-urlencoded", \
"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36", \
"Origin":"https://somedomain.com", \
"X-Requested-With":"XMLHttpRequest", \
"Connection":"keep-alive", \
"Accept-Language":"en-US,en;q=0.8", \
"Referer":"https://somedomain.com/release_cr_new.html?releaseid=%s&v=2&m=a&prev_release_id=%s" % (current_release_id, previous_release_id), \
"Host":"somedomain.com", \
"Accept-Encoding":"gzip,deflate,sdch", \
"Cookie":'cookie_val'}

for bug_id in ids:
  print bug_id
  data = {'dump_json':'1','releaseid':current_release_id, 'v':'2','m':'a','prev_release_id': previous_release_id,'bug_ids': bug_id, 'set_cols':'sqa_status&sqa_updates%5B0%5D%5Bbugid%5D=' + bug_id + '&sqa_updates%5B0%5D%5Bsqa_status%5D=6'}
  print 'current_release_id' , data['releaseid']
  print 'previous_release_id', data['prev_release_id']
  r = requests.post(post_url, data=json.dumps(data), headers=headers2)
  print r.text

The output I'm getting is a pretty generic html message that I've seen before when I've queried the server in the wrong way. So I know I'm reaching the right server at least.

I'm not really expecting any output. This should just post to the server and update a field in the DB.

Community
  • 1
  • 1
Ramy
  • 20,541
  • 41
  • 103
  • 153
  • Can you post an example (including input, expected output, and actual output)? You might point at http://httpbin.org/ (you can fork on github if you don't want to send your data there). That might help you in debugging the differences. – Wayne Werner Sep 27 '13 at 18:37
  • I tried to update it with an example. I was hesitant to do so at first because I"m notoriously bad at leaving something in that I should mask out. – Ramy Sep 27 '13 at 18:50
  • i played around a bit with the encoding - removing it altogether with No change in outcome. – Ramy Sep 27 '13 at 18:59

2 Answers2

5

Anatomy of an http response

Example (loading this page)

HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: Fri, 27 Sep 2013 19:22:41 GMT
Last-Modified: Fri, 27 Sep 2013 19:21:41 GMT
Vary: *
X-Frame-Options: SAMEORIGIN
Date: Fri, 27 Sep 2013 19:21:41 GMT
Content-Length: 12706

<!DOCTYPE html>
<html>
... truncated rest of body ...
  1. The first line is the status line and consists of the status code and status text.
  2. Headers are key/value pairs. Headers are ended with an empty new line. The empty line denotes there are no more headers and the start of the payload / body follows.
  3. body consumes the rest of the message.

The following explains how to extract the 3 parts:

Status Line

Use the following to get the status line sent back from the server

>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404

>>> bad_r.raise_for_status()
Traceback (most recent call last):
  File "requests/models.py", line 832, in raise_for_status
    raise http_error
requests.exceptions.HTTPError: 404 Client Error

(source)

Headers:

r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
# response headers: 
r.headers
# request headers:
r.request.headers

Body

Use r.text.

Post Request Encoding

The 'content-type' you send to the server in the request should match the content-type you're actually sending. In your case, you are sending json but telling the server you're sending form data (which is the default if you do not specify).

From the headers you show above:

"Content-Type":"application/x-www-form-urlencoded",

But your request.post call sets data=json.dumps(data) which is JSON. The headers should say:

"Content-type": "application/json",

Jonathan
  • 5,736
  • 2
  • 24
  • 22
  • this is great! So, I just copied the `json.dumps` piece blindly. The form-urlencoded is the correct encoding type. I changed this to `data=data` and now the response seems more accurate but it's not quite doing what I expect. I'll accept this answer do more digging. – Ramy Sep 27 '13 at 19:43
0

The value returned from the request object contains the request information under .request.

Example:

r = requests.request("POST", url, ...)

print("Request headers:", r.request.headers)
print("Request body:", r.request.body)

print("Response status code:", r.status_code)
print("Response text:", r.text.encode('utf8'))
aled
  • 21,330
  • 3
  • 27
  • 34