How to handle multiple request for a MIDDLEWARE in SCRAPY (captchas y multiple retries)

Question

I'm trying to build a spider who breaks a dynamic captcha with just Scrapy, i have done it BUT of course when i'm breaking the captcha is not always correct, so I HAVE TO make it retry multiple times (max. 10) to really enter the 'login' page for MULTIPLE ACCOUNTS (btw i'm using PIL and pytesseract to handle the captcha). I have read How do I set up Scrapy to deal with a captcha it's a great help BUT it gives me this error:

TypeError: process_response() got multiple values for argument 'request'

This is my spider code:

#self.crawler.settings['MY_ACCOUNTS']= i have defined my list of accounts with "keys"

def start_requests(self):
    requests = []

    for i, account in enumerate(self.crawler.settings['MY_ACCOUNTS']):
        request = scrapy.Request(same_url, callback = self.parse, meta={'key':account['key']} ,dont_filter=True)
        request.meta['cookiejar'] = i
        requests.append(request)
        yield request

#I HAVE TO USE MULTIPLE REQUEST JUST TO GET THE CORRECT SESSION FOR THE COOKIES AND GET THE IMAGE, IF NOT THE IMAGE ALWAYS CHANGE
def parse(self, response):
    key= response.meta.get('key')
    sessionid = response.headers.getlist('Set-Cookie')[0].decode().split(';')[0]
    cookien = sessionid.split('=')[0].strip()
    sessionid = sessionid.split('=')[1].strip()
    yield scrapy.Request(same_url, meta = {'cookieN': cookien, 'cookieV' : sessionid, 'key':key,'cookiejar': response.meta['cookiejar']},cookies = {cookien :sessionid}, callback = self.parse_with_cookies, dont_filter=True)

#HERE I'M TRYING TO BREAK THE CAPTCHA FILLING A FORM
def parse_with_cookies(self, response):
    #break captcha (get text)

    data = {'key':key,
    'catcha':text
    }#fill with the data

    yield FormRequest.from_response(response=response,formname="mainForm", formdata=data, callback = self.parse_each_key,meta = {'solve_captcha': True,'CookieName':cookieN,'CookieValue':cookieV , 'key':key,'cookiejar': response.meta['cookiejar'],'catpcha_retries':0},cookies={cookieN:cookieV},dont_filter = True)

#the status code is always 200 even if the captcha is not broken so i search for words if it's incorrect
def parse_each_key(self, response):
    if ("incorrect" not in response.text) and ("error" not in response.text):
         #extract what i want
    else:
         self.logger.error("Login failed")

-----MIDDLEWARE---

#i haven't change to much from the link, i know that i have to fill the functions find_catpcha and solve_captcha but it doesn't even get there

class CaptchaMiddleware(object):
max_retries = 5
def process_response(request, response, spider):
    if not request.meta.get('solve_captcha', False):
        return response  # only solve requests that are marked with meta key
    catpcha = find_catpcha(response)
    solved = solve_captcha(captcha)
    if solved:
        response.meta['catpcha'] = captcha
        response.meta['solved_catpcha'] = solved
        return response
    else:
        # retry page for new captcha
        # prevent endless loop
        if request.meta.get('catpcha_retries', 0) == 5:
            logging.warning('max retries for captcha reached for {}'.format(request.url))
            raise IgnoreRequest 
        request.meta['dont_filter'] = True
        request.meta['captcha_retries'] = request.meta.get('captcha_retries', 0) + 1
        return request

I don't know if it's because i'm sending multiple request at the same time or maybe something in my code.

Thanks

score 0 · Answer 1 · answered Jan 16 '19 at 12:05

You are missing the self parameter in the definition of process_response() in your middleware. That is the cause of the exception.

If you replace process_response(request, response, spider) with process_response(self, request, response, spider) you should stop getting that exception.

How to handle multiple request for a MIDDLEWARE in SCRAPY (captchas y multiple retries)

1 Answers1