scrapy throws error 500 after sending Post request

Question

I can successfully send Post request and receive all data through Postman. Which returns this python code for export

import requests

url = "https://cl.ingrammicro.com/_layouts/CommerceServer/IM/SearchService.svc/Search"

payload = "{\"request\":{\"State\":\"PNavDS=N:0\",\"Keywords\":[],\"Page\":0,\"PageLayout\":0,\"Mode\":12,\"Term\":\"0\",\"ExchangeRate\":null}}"
headers = {
    'origin': "https://cl.ingrammicro.com",
    'user-agent': "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
    'content-type': "application/json",
    'accept': "text/plain, */*; q=0.01",
    'cache-control': "no-cache",
    'postman-token': "e4a3a94a-fa02-5ea3-2386-295a9fe21f3b"
    }

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

This post request returns neat json dict.

When I use the same in scrapy I got 500 error. Here is my Scrapy code.

url = "https://cl.ingrammicro.com/_layouts/CommerceServer/IM/SearchService.svc/Search"
            headers = {
                    'origin': "https://cl.ingrammicro.com",
                    'user-agent': "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
                    'content-type': "application/json",
                    'accept': "text/plain, */*; q=0.01",
                    'cache-control': "no-cache",

                    }

            payload = {"request":{"State":"PNavDS=N:0","Keywords":[],"Page":0,"PageLayout":0,"Mode":12,"Term":"0","ExchangeRate":'null'}}

            return FormRequest(url, method = "POST", body=json.dumps(payload),headers = headers, callback=self.parse_data)

Scrapy error.

2017-07-31 16:59:35 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <POST https://cl.i
ngrammicro.com/_layouts/CommerceServer/IM/SearchService.svc/Search> (failed 3 times): 500 Internal
Server Error
2017-07-31 16:59:35 [scrapy.core.engine] DEBUG: Crawled (500) <POST https://cl.ingrammicro.com/_lay
outs/CommerceServer/IM/SearchService.svc/Search> (referer: None)

I also tried to pass this as a string( as Postman code suggests) but the error is the same)

url = "https://cl.ingrammicro.com/_layouts/CommerceServer/IM/SearchService.svc/Search"
            headers = {
                    'origin': "https://cl.ingrammicro.com",
                    'user-agent': "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
                    'content-type': "application/json",
                    'accept': "text/plain, */*; q=0.01",
                    'cache-control': "no-cache",

                    }
            payload = "{\"request\":{\"State\":\"PNavDS=N:0\",\"Keywords\":[],\"Page\":0,\"PageLayout\":0,\"Mode\":12,\"Term\":\"0\",\"ExchangeRate\":null}}"
            q = {"request":{"State":"PNavDS=N:0","Keywords":[],"Page":0,"PageLayout":0,"Mode":12,"Term":"0","ExchangeRate":"null"}}
            p={}
            p['request'] = str(q['request'])
            p['request']= p['request'].replace("'",'"')



            return FormRequest(url, method = "POST", body=urlencode(q) ,headers = headers, callback=self.parse_data)

Where am I wrong?

Why do you want scrapy for this, if you are sure you can do it using **requests**? — Nabin, Jul 31 '17 at 14:36
It has 600 pages, which is long enough, als this is 1 domain out of 3 for the same project with same item structure. So Scrapy is my choice # 1. — Billy Jhon, Jul 31 '17 at 14:42
Have you tried to pass `payload` to `formdata` argument of `FormRequest` instead of using `body`? — Tomáš Linhart, Jul 31 '17 at 16:57

score 0 · Accepted Answer · answered Sep 25 '18 at 10:28

0

What I did was replacing parts of the code with other parts. But correct solution would be to use json.dumps(payload) which will give us exactly what we need to proceed and allows to manipulate dict as expected. P.S. thanks to this answer - by @zeekay

answered Sep 25 '18 at 10:28

Billy Jhon

1,035
15
30

scrapy throws error 500 after sending Post request

1 Answers1