POST request works in Postman, but not in Python Requests (200 response with robot detection)

Question

I have a POST request that works perfectly with both Postman an cURL (it returns a JSON blob of data). However, when I perform the exact same request with Python's Requests library, I get a 200 success response, but instead of my JSON blob, I get this:

<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>

I've used HTTP request bins to verify that the request from Postman/cURL is exactly the same as the one from Python Requests.

Here is my Postman request in cURL:

curl -X POST \
  https:/someurl/bla/bla \
  -H 'Content-Type: application/json' \
  -H 'Postman-Token: 2488e914-531e-4ac7-ae8d-8490b2242396' \
  -H 'Referer: https://www.host.com/bla/bla/' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0' \
  -H 'cache-control: no-cache' \
  -d '{"json1":"blabla","etc":"etc"}'

...and here is my Python code:

payload = {
      "json1": "blabla",
      "etc": "etc",
    }

    headers = {
        'Host': 'www.host.com',
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
        'Accept': 'application/json, text/javascript, */*; q=0.01',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Referer': 'https://www.host.com/bla/bla/', 
        'Content-Type':'application/json',
        'X-Requested-With': 'XMLHttpRequest',
        'Connection': 'keep-alive',
        'Origin': 'https://www.host.com',
    }

    s = requests.Session()
    response_raw = s.post(url, json=payload, headers=headers)
    print(response_raw)
    print(response_raw.text)

I have verified that the payload and headers are correct and valid. Any help would be much appreciated; thanks!

Try turning on debugging and checking the actual request being made... `from http.client import HTTPConnection; HTTPConnection.debuglevel = 1` — AChampion, Feb 26 '19 at 05:17
Where is your `Postman-Token: 2488e914-531e-4ac7-ae8d-8490b2242396` in python code? You have it in `curl` but not in python. — Matej, Feb 26 '19 at 22:54
@Matej that was automatically added by Postman -- I removed it from the config, and it still works fine. I updated my original question to state that the HTTP requests for both Postman and Python Requests ate the same, as measured by what HTTP request bins received :) — woodenstick, Feb 27 '19 at 00:17
@Matej sure. Here's the [cURL request](https://pastebin.com/Gft5J419), and here's the [Python code](https://pastebin.com/kEPfMpLT). The payload data has been omitted since it's not necessary for the question -- the server still responds very differently to each of these requests... — woodenstick, Feb 28 '19 at 00:52
@woodenstick I got same response (code - 400) from python as from curl. — Matej, Feb 28 '19 at 09:09
@Matej yes, the response code is the same, but the actual response is different. Curl returns json data, whereas Python returns HTML stating that Incapsula blocked the request. — woodenstick, Feb 28 '19 at 20:12
@woodenstick Yeah, I tried it again and on the mobile phone (with temrux) I got same output as curl but on computer I got different, weird, I will try to find where is the difference. — Matej, Feb 28 '19 at 20:16
@woodenstick I tried catch request from mobile and from computer. Bot were same. I am really confused. — Matej, Feb 28 '19 at 20:57

venkat · Answer 1 · 2020-05-07T10:38:53.853

2

You are getting a 200 success response but not JSON data in the response.
This means that is just a response object. It contains only response code
to extract blob information from the response, convert response object to json
simply json_resp = response_raw.json()
This json_resp contains your actual response details.

edited May 07 '20 at 10:38

answered May 06 '20 at 12:00

venkat

453
4
16

This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - [From Review](/review/low-quality-posts/26046733) – Kvam May 07 '20 at 10:15
1

@Kvam That is the answer which works. If add that in the comment, that might be ignored for those who fall here. I suggest you to please read the question clearly and what is the solution author expecting. – venkat May 07 '20 at 10:29

score 1 · Answer 2 · answered May 01 '19 at 17:15

I had a similar issue that I was able to resolve by sending a cookie in the request. Try this:

...
my_cookie = {"Cookie": "cookie text..."}

s = requests.Session()
response_raw = s.post(url, json=payload, headers=headers, cookies=my_cookie)
print(response_raw)
print(response_raw.text)
print(response_raw.content)

You can grab the cookie from the Network tab in the browser's Dev Tools console in the Request Headers section. It sounds like you may also be able to get the cookie using Python's CookieJar lib.

score 0 · Answer 3 · answered Aug 10 '21 at 06:17

0

Try passing the verify option as false.

requests.get('https://example.com', verify=False)

answered Aug 10 '21 at 06:17

Aditya Goel

201
1
15

POST request works in Postman, but not in Python Requests (200 response with robot detection)

3 Answers3

Linked