To scrape Google Search you can use only Beautifulsoup webscraping library without selenium webdriver that will increase the speed of the script.
To avoid blocks from Google, if using requests could be to rotate user-agent, for example, to switch between PC, mobile, and tablet, as well as between browsers e.g. Chrome, Firefox, Safari, Edge and so on, as default user-agent in requests library is a python-requests so the website can understand that it's a script that sends a request.
To collect the necessary information (email, description, title, number, etc.) you can use CSS selectors search which are easy to identify on the page using a SelectorGadget Chrome extension (not always work perfectly if the website is rendered via JavaScript).
import requests, re, json, lxml
from bs4 import BeautifulSoup
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
}
params = {
'q': 'Facebook.com Dantist gmail.com', # query
'hl': 'en', # language
'gl': 'us' # country of the search, US -> USA
}
html = requests.get(f'https://www.google.com/search',
headers=headers,
params=params).text
soup = BeautifulSoup(html, 'lxml')
data = []
for result in soup.select('.tF2Cxc'):
title = result.select_one('.DKV0Md').text
link = result.find('a')['href']
snippet = result.select_one('.lyLwlc').text
match_email = re.findall(r'[\w\.-]+@[\w\.-]+\.\w+', snippet)
email = ''.join(match_email)
# https://stackoverflow.com/a/3868861/15164646
match_phone = re.findall(r'((?:\+\d{2}[-\.\s]??|\d{4}[-\.\s]??)?(?:\d{3}[-\.\s]??\d{3}[-\.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-\.\s]??\d{4}|\d{3}[-\.\s]??\d{4}))', snippet)
phone = ''.join(match_phone)
data.append({
'Title': title,
'Link': link,
'Email': email if email else None,
'Phone': phone if phone else None
})
print(json.dumps(data, indent=2, ensure_ascii=False))
Example output:
[
{
"Title": "Island Dental Associates | Franklin Square NY - Facebook",
"Link": "https://www.facebook.com/IslandDentalAssociates/",
"Email": "islanddentalassociatesny@gmail.com",
"Phone": "(516) 271-0585"
},
{
"Title": "Dental Bright | Houston TX - Facebook",
"Link": "https://www.facebook.com/DentalBrightHouston/",
"Email": "Dentalbrighttx@gmail.com",
"Phone": "(713) 783-6060"
},
# ...
]
As an alternative, you can use Google Search Engine Results API from SerpApi. It's a paid API with a free plan.
The difference is that it will bypass blocks (including CAPTCHA) from Google, no need to create the parser and maintain it.
Code example:
from serpapi import GoogleSearch
import os, json, re
params = {
"engine": "google", # search engine. Google, Bing, Yahoo, Naver, Baidu...
"q": "Facebook.com Dantist gmail.com", # search query
"api_key": os.getenv('API_KEY') # your serpapi api key
}
search = GoogleSearch(params) # where data extraction happens
results = search.get_dict() # JSON -> Python dictionary
data = []
for result in results['organic_results']:
title = result['title']
link = result['link']
snippet = result['snippet']
match_email = re.findall(r'[\w\.-]+@[\w\.-]+\.\w+', snippet)
email = '\n'.join(match_email)
match_phone = re.findall(r'((?:\+\d{2}[-\.\s]??|\d{4}[-\.\s]??)?(?:\d{3}[-\.\s]??\d{3}[-\.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-\.\s]??\d{4}|\d{3}[-\.\s]??\d{4}))', snippet)
phone = ''.join(match_phone)
data.append({
'title': title,
'link': link,
'email': email if email else None,
'phone': phone if phone else None
})
print(json.dumps(data, indent=2, ensure_ascii=False))
Output:
The answer is identical to the answer bs4.
