5

Im trying to solve a captcha using 2Captcha.com service. First I'm saving the captcha image:

urllib.urlretrieve(captcha_image_link, 'captcha.jpg')

Next I need to upload the image to the server to recognize it. Using plain requests its just as simple as 

files = {'file': open('captcha.jpg', 'rb')}
payload = {'key': TWOCAPTCHA_APIKEY, 'method': 'post'}
request = requests.post('http://2captcha.com/in.php', files=files, data=payload)

But how to make the same request with Scrapy? I mean, how to attach an image file to a POST Request? Is this possible? If not, then I would like to know whether its too bad idea to use plain POST request (and urlretrieve as well) inside a Scrapy spider or not?

sky
  • 121
  • 8
  • Why must you make the request with scrapy. In this case you just want the response of the solved captcha so the next request can be made. So `import requests` – tread Aug 13 '17 at 22:42
  • I just want to clarify it for myself - Is it safe and correct to use `requests` inside a Scrapy spider? Would plain request slow it down? – sky Aug 13 '17 at 23:02
  • It might do. You can read this interesting article but I can't speak on whether it is `right` or `wrong`: http://www.scrapinginsider.com/2016/01/scrapy-urllib2-requests-beautifulsoup-lxml.html – tread Aug 14 '17 at 11:13
  • Thanks for the link. Though that article offers me to scrape the website with BS and requests running in some threads - which is not that Im looking for – sky Aug 14 '17 at 15:54
  • Possible duplicate of [Scrapy upload file](https://stackoverflow.com/questions/39303851/scrapy-upload-file) – Gallaecio Jan 22 '19 at 13:58

1 Answers1

0

You can send POST request using Scrapy (see documentation), but there are two things to consider:

  1. You would have to encode the files yourself while requests.post does this for you.
  2. Scrapy would send the request asynchronously, which is probably not what you want when you need to solve captcha.

Thus, I see nothing wrong using your current aproach inside Scrapy spider.

Tomáš Linhart
  • 9,832
  • 1
  • 27
  • 39