Python requests not responding

Question

Done a few small successful projects, been struggling to get the requests from this website fro ages - any tips?

UPDATE - Would like to get full beautiful soup request so I can start scraping the information from the tables

from bs4 import BeautifulSoup
import requests

r = requests.get("http://www.transfermarkt.co.uk/championship/marktwerte/wettbewerb/GB2")
soup = BeautifulSoup(r.content,"html.parser")
print soup

returning

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</hr></body>
</html>

what do you want it to return? (update your question!) (and welcome to SO!) — user1269942, Dec 23 '15 at 20:28

score 5 · Accepted Answer · answered Dec 23 '15 at 20:29

5

You need to pretend to be a real user with a browser and provide a User-Agent header:

r = requests.get("http://www.transfermarkt.co.uk/championship/marktwerte/wettbewerb/GB2", headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
})

Demo:

>>> from bs4 import BeautifulSoup
>>> import requests
>>> 
>>> r = requests.get("http://www.transfermarkt.co.uk/championship/marktwerte/wettbewerb/GB2", headers={
...     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
... })
>>> soup = BeautifulSoup(r.content,"html.parser")
>>> print(soup.title.get_text())
Top market values 15/16 - Championship - Transfermarkt

answered Dec 23 '15 at 20:29

alecxe

462,703
120
1,088
1,195

Excellent, thank you very much introducing me to user agents - will mark solved shortly – PythonNoob12345 Dec 23 '15 at 20:33
nice..beat me by a whisker! – user1269942 Dec 23 '15 at 20:39
I had the exact same problem with the same site, does it matter what kind of user agent I provide? – PEREZje Jun 02 '18 at 18:40
If you are still facing the same issue, check this post https://stackoverflow.com/questions/51154114/python-request-get-fails-to-get-an-answer-for-a-url-i-can-open-on-my-browser/51154676 – Utkarsh Bhushan Aug 12 '20 at 16:23

score 1 · Answer 2 · answered Jun 23 '21 at 11:50

There are some sites where requests fail to give response as many of them track if the request originating party is a browser or a bot. So, let us look like a browser.
It can be done by modifying the header as follows:

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36', "Upgrade-Insecure-Requests": "1","DNT": "1","Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language": "en-US,en;q=0.5","Accept-Encoding": "gzip, deflate"}

And then, just simply add this header your GET request as follow:

response = requests.get("https://example.com",headers=headers)

In total you will get:

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36', "Upgrade-Insecure-Requests": "1","DNT": "1","Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language": "en-US,en;q=0.5","Accept-Encoding": "gzip, deflate"}
response = requests.get("https://example.com",headers=headers)

Python requests not responding

2 Answers2