python request get a REST and return 404

Question

I have a REST url like:

url='http://xx.xx.xx.xx/server/rest/line/125'

I can get correct return json by command curl, like this:

curl -i http://xx.xx.xx.xx/server/rest/line/125

But when i use Python3 requests, it allways return 404, Python code:

import requests
resp = requests.get(r'http://xx.xx.xx.xx/server/rest/line/125')
print(resp.status_code)

Anyone can tell me the problem? Is the server block the request?

[404 means you have an invalid URL](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes). — IMCoins, Jan 02 '18 at 13:13
I konw 404 means, but i sure the URL in curl command is same to Python code. So i don't know why. — sonox, Jan 02 '18 at 13:16
Neither do we, if that's all there is to go on. You'll have to do some more troubleshooting... what headers is `curl` sending? What happens if you send the same headers with `requests.get()`? — glibdud, Jan 02 '18 at 13:35
@sonox, resp = requests.get(r'http://xx.xx.xx.xx/server/rest/line/125'), why is there an 'r' in the get method ? — Hassan Abbas, Jan 02 '18 at 13:42
@glibdud I dont send headers with -H in `curl` command, and no headers in `requests.get()`. Is there any way can view actual request head and content from `requests.get()`? — sonox, Jan 02 '18 at 13:49
Try [How can I see the entire HTTP request that's being sent by my Python application?](https://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application) and [How can I see the request headers made by curl when sending a request to the server?](https://stackoverflow.com/questions/866946/how-can-i-see-the-request-headers-made-by-curl-when-sending-a-request-to-the-ser). — glibdud, Jan 02 '18 at 13:54
@glibdud I compared the header of `curl` and `requests.get()`, and find the different is User-Agent. When i use `curl` User-Agent in requests' headers, its work right. So i think maybe the server do something to block Python spder. Finally, thank you all guys! — sonox, Jan 02 '18 at 14:35

Niharika Bitra · Answer 1 · 2019-02-08T20:27:00.700

A 404 is displayed when:

The URL is incorrect and the response is actually accurate.
Trailing spaces in the URL
The website may not like HTTP(S) requests coming from Python code. Change your headers by adding "www." to your Referer url.

resp = requests.get(r'http://www.xx.xx.xx.xx/server/rest/line/125')

or

headers = {
       'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
    }
result = requests.get('https://www.transfermarkt.co.uk', headers=headers)

In your case, try option #3.

python request get a REST and return 404

1 Answers1