-2

I would like to scrap/extract the number of results given by Google Search Engine for a given word with Python, but I can't with Beautifulsoup and Request library. If anyone can help me out that would be great.

A print screen is attached to better explain the number I want to extract.

Google Search Engine Result 'decoration'

Hervé Ramboasalama
  • 1
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community May 29 '22 at 02:29

1 Answers1

0

If you inspect the search page, you can see that the value is inside a div with an id. inspect result

That is good for your purpose since ids uniquely identify elements within the page. To find out how to get an element by its id you just need to make a google search (first result), and then you can get the text from the "text" property. You will also need to parse the text, to only extract the number.

Edit: Looks like, without providing a user agent, the google API will not return the full page. If you send the "User-Agent" header with the value from your browser, it should work. A quick way to check yourself is to run the request in Postman, where you can just search if the result has what you need.

html = BeautifulSoup(requests.get(
            'https://www.google.com/search?q=f', headers={ "user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'}).text, features="html.parser")

text = html.body.find('div', attrs={'id': 'result-stats'}).text

print(text)

Also, it's worth mentioning that Google provides endpoints exactly for this kind of purpose. Here's another question referring to the same problem (link).

Matei Macoveiciuc
  • 179
  • 1
  • 12