Sending a POST with mechanize and requests.

Question

I am trying to send a POST using mechanize however my code won't work sometimes(and I know why). I have used mechanize, twill and requests. With mechanize and twills it is working and with requests it is not. Probably I am doing it in wrong way.

My mechanize code. The following works :

#!/usr/bin/env python
import sys
import urllib
import mechanize
from mechanize import ParseResponse, urlopen, urljoin
response = urlopen(url)
forms = ParseResponse(response, backwards_compat=False)
form = forms[0]
form["username"] = "avi"
form["password"] = "stackoverflow"
urlopen(form.click())

My twill code. It also works :

import os
import twill
from twill.commands import *
out = open(os.devnull,"w")
twill.set_output(out)
go(url)
formvalue("1", "username", "avi")
formvalue("1", "password", "stackoverflow")
submit()
go(url2) #some protected page
content = show()
print 'content is',content[:100]

From what I understand, mechanize or twill first gets the page, fills the form and sends the form. But problem here is, sometimes target page displays random page, without any form. In that case I get an error, obviously because there is no form to show. I don't want to handle this error, because I already know the post URL. even though some random page is displayed, after I click next, same form is loaded. The POST url, fields all remain same. So I wanted to send the POST request directly, since I already know details everything required. Here is my code, based on :

browser = mechanize.Browser()
parameters = {"username" : "avi",
          "password" : "stackoverflow",
         }  
data = urllib.urlencode(parameters)
browser.open(post_url,data)
cool = browser.open(post_url + '%s' % data).read()
print cool

I get an error :

urllib2.URLError: <urlopen error [Errno 61] Connection refused>

I tried same thing with requests. But I am getting same error as above. Here is the code :

import requests
from requests import session
payload = { 'username': 'avi','password': 'stackoverflow'}
url1 = 'http://example.com/login.php'
url2 = 'http://example.com/protected.php'

with session() as c:
    c.post(url1, data=payload)
    c.get(url2)

So I poked here and there, read few more code available online and I think it is not working because to prevent CSRF. So I went to same page with twill, did showforms and saw there is some value with token :

Here is what I want :

1. Do POST with mechanize and requests (without downloading the page first)
2. How to handle CSRF tokens ?
3. How to debug 'urlopen error [Errno 61] Connection refused'

Answer 1

I've used the code below with success:

params = {u'user_login': self.USER, u'password':self.PASSWORD}
data = urllib.urlencode(params)
request = mechanize.Request( loginURL )
response = mechanize.urlopen(request, data=data)

To handle the CSRF token issue, I added the line below to my controller:

skip_before_filter :verify_authenticity_token, :only => [:create]

To keep the session between requests, I use a cookiesJar. But you can retrieve the CSRF token from your request and keep it in your application, adding it the each new request.

Answer 2

You have to retrieve the form page in order to get the session cookies and the csrf token.

The connection refused error might be due you got blocked by the site due to posting to the login page through automated means.