Scrapy: Do form login and then work with that session

Question

I'm trying to do the following:

1. log into a web page (in my case zendesk.com)
2. use that session to do some post requests

In fact zendesk misses some apis (create/alter macros) which I now need to simulate simulating a browser session.

So I'm not writing a spider but try to interact with the website as my script proceeds. The post requests are not known from the start but only during my script.

In the Scrapy docs, there is the following example to illustrate how to use an authenticated session in Scrapy:

class LoginSpider(BaseSpider):
  name = 'example.com'
  start_urls = ['http://www.example.com/users/login.php']

  def parse(self, response):
    return [FormRequest.from_response(response,
                formdata={'username': 'john', 'password': 'secret'},
                callback=self.after_login)]

  def after_login(self, response):
    # check login succeed before going on
    if "authentication failed" in response.body:
        self.log("Login failed", level=log.ERROR)
        return

    # continue scraping with authenticated session...

But it looks like this only works for scraping, but in my case I just want to "hold" the session and further work with that session. Is there a way to achieve this with scrapy, or are there tools that better fit this task?

Answer 1

Thanks a lot @wawaruk. Based on the stackoverflow post you linked that's the solution I came up with:

import urllib, urllib2, cookielib, re

zendesk_subdomain = 'mysub'
zendesk_username = '...'
zendesk_password = '...'

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
resp = opener.open('http://%s.zendesk.com/access/unauthenticated' % (zendesk_subdomain))
s = resp.read()
data = dict()
data['authenticity_token'] = re.findall('<input name="authenticity_token" type="hidden" value="([^"]+)"', s)[0]
data['return_to'] = 'http://%s.zendesk.com/login' % zendesk_subdomain
data['user[email]'] = zendesk_username
data['user[password]'] = zendesk_password
data['commit'] = 'Log in'
data['remember_me'] = '1'

opener.open('https://localch.zendesk.com/access/login', urllib.urlencode(data))

from there with opener all pages can be accessed, e.g.

opener.open('http://%s.zendesk.com/rules/new?filter=macro' % zendesk_subdomain)