scrapy - AjaxMethod not available

Question

I am trying to practice learning Scrapy web crawling and using a classified car site for the subject in order to check into countermeasures. I know that the X-AjaxPro-Method exists because Chrome Developer Tools shows the headers being passed and proper response received. But when done in Scrapy shell I get "This method is either not marked with an AjaxMethod or is not available."

Here are the shell commands used:

>>> from scrapy.http import FormRequest

>>> 
request=FormRequest(url='https://www.carwale.com/ajaxpro/CarwaleAjax.AjaxClassifiedBuyer,Carwale.ashx',headers={"X-AjaxPro-Method":"ProcessUsedCarPurchaseInquiry","Content-Type":"application/x-www-form-urlencoded; charset=UTF-8","X-Requested-With":"XMLHttpRequest"},formdata={"profileId":"D1249107","buyerName":"","buyerEmail":"","buyerMobile":"9938223299","carModel":"","makeYear":"","pageUrl":"https://www.carwale.com/used/cars-in-karnal/chevrolet-enjoy-d1249107/?rk","isP":"False","transToken":"","ltsrc":"","buyerSourceId":"4","comments":"","cwc":"buJNfItyQKBP8a3OahoJsOOmg","utma":"\"52149691.1076750176.1492103717.1492447801.1492447801.8\"","utmz":"\"52149691.1492103720.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)\"","originId":"3","isFromCaptcha":"","isGSDClick":"","isRecommended":"","isCertificationDownload":""})

>>> fetch(request)
2017-04-18 08:45:32 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.carwale.com/ajaxpro/CarwaleAjax.AjaxClassifiedBuyerCarwale,Carwale.ashx> (referer: None)

>>> print(response.body)
{"error":{"Message":"This method is either not marked with an AjaxMethod or is not available.","Type":"System.NotSupportedException"}}

>>>

The original page is at https://www.carwale.com/used/cars-in-karnal/chevrolet-enjoy-d1249107/?rk=69&isP=false and a mobile phone number has to be entered in order to get "Seller Details."

So, I have dug a little further and will share some more information. I have been able to export the XHR as a curl command using the developer tools in the browser and then trimmed this down so that it appears to me that the only required header is the X-AjaxPro-Method because the curl command works with just that header and the data.

Also got it to work using the Python requests library.

It seems your request does not identify itself as the page is expecting. You might want to copy the curl request from Chrome (inspect - network) and recreate this in Scrapy. If you put the original link + actions how to fire the ajax script, we can try to help you out as well. — Casper, Apr 18 '17 at 13:53

score 1 · Accepted Answer · answered Apr 22 '17 at 12:17

Comparing your posted request data with what I see in Firebug, I suspect at least one of these is missing in your request:

Cookie data (should be handled by scrapy if you turned on cookies in settings.py and made scrapy visiting https://www.carwale.com/used/cars-in-karnal/chevrolet-enjoy-d1249107/?rk=69&isP=false first )
User Agent header (set USER_AGENT in settings.py to something that looks like a real browser e.g. 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0' )
Referer header (should be handled automatically by scrapy if you visited https://www.carwale.com/used/cars-in-karnal/chevrolet-enjoy-d1249107/?rk=69&isP=false before sending the request )

All in all such ajax powered sites as carwale.com have a lot of moving parts and are a not so good object to "start learning scrapy"

PS: a better way to use FormRequest is to do request = FormRequest.from_response(response_with_form_page, ...). This works well on most forms as scrapy will extract all the hidden POST parameters automatically from the form page. For details see: https://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.FormRequest.from_response

I was able to get this to work using a scrapy Request rather than a FormRequest. Thanks for the good advice. — jcaster, Apr 26 '17 at 17:07

scrapy - AjaxMethod not available

1 Answers1