How to collect the keywords from the search results using Selenium Python

Question

I want to creat a automation go to https://keywordtool.io/ and search for an topic and collect the keyword from it.

Code trials:

from multiprocessing.connection import wait
import time
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

Path = "C:\Program Files\chromedriver.exe"
##browser=webdriver.Chrome(ChromeDriverManager().install())
browser= webdriver.Chrome(Path)
browser.get('https://keywordtool.io')

search = browser.find_element(By.ID, 'search-form-google-keyword-md')
search.send_keys("funny shirt")
search.send_keys(Keys.RETURN) # hit return after you enter search text
wait(10)

What is the problem here?

Answer 1

There are several issues here:

1. You need to wait for the page to be loaded before applying this line

search = browser.find_element(By.ID, 'search-form-google-keyword-md')

The most recommended way to do that is to use WebDriverWait expected_conditions explicit waits.
2) with your current code this line wait(10) will cause exception since wait object still not defined.
3) if you want the browser to stay opened you can use the 'detach' option when starting chromedriver as following:

import time
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)

path = "C:\Program Files\chromedriver.exe"
browser= webdriver.Chrome(executable_path = path, options = chrome_options)
wait = WebDriverWait(browser, 20)

browser.get('https://keywordtool.io')

search = wait.until(EC.visibility_of_element_located((By.ID, "search-form-google-keyword-md")))

search.send_keys("funny shirt")
search.send_keys(Keys.RETURN) # hit return after you enter search text

Answer 2

How I understood your Question you want that the browser keeps open 10 seconds after the enter. When that is what you want you have to use time.sleep(10) instead of wait(10).

Answer 3

To extract the Keywords from the search results you have to induce WebDriverWait for presence_of_all_elements_located() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  driver.get('https://keywordtool.io/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='keyword']"))).("funny shirt" +Keys.RETURN)
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table.search-results-table tr span.font-weight-bold")))])
  

• Using XPATH:

  driver.get('https://keywordtool.io/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='keyword']"))).send_keys("funny shirt" +Keys.RETURN)
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.XPATH, "//table[contains(@class, 'search-results-table')]//tr//span[@class='font-weight-bold']")))])
  

• Console Output:

  ['funny shirt', 'funny shirts', 'funny shirts for men', 'funny shirts for women', 'funny shirt sayings', 'funny shirt ideas', 'funny shirts for dads', 'funny shirt sayings for adults', 'funny shirt quotes', 'funny shirt designs', 'funny shirt svg', 'funny shirt jokes', 'cool funny shirt twitter', 'funny shirt for dad', 'funny shirt sayings for guys', 'funny shirt amazon', 'funny shirt ad', 'funny shirts australia', 'funny shirts about dogs', 'funny shirts about covid', 'funny shirts about drinking', 'funny shirts about work', 'funny shirts about wife', 'funny shirts about running', 'funny shirts about wine', 'amazon funny shirt', 'adhd t shirt funny', 'funny football shirt names and numbers', 'funny big sister announcement shirt', 'a funny shirt saying', 'funny names for the back of a shirt', 'funny for a girl shirt', 'funny things to put on a shirt', 'funny shirt brands', 'funny short bio', 'funny shirt business names', 'funny shirt baby', 'fun shirt brooks brothers', 'funny shirts brisbane', 'funny shirts beer', 'funny shirts bff', 'funny shirts bride', 'funny shirts batman', 'buckin funny shirt co', 'blink 182 funny shirt', 'blm funny shirt', 'buffalo bills funny shirt', 'bmw funny shirt', 'baby yoda funny shirt', 'bts funny shirt', 'best funny short jokes', 'best friend funny shirt design', 'best funny shirt designs', 'funny shirt company', 'funny shirt company names', 'funny short clips', 'funny short captions', 'funny short comics', 'funny short cowboy quotes', 'funny short campfire stories', 'funny short characters', 'funny short christian jokes', 'funny short captions for instagram', 'clowns are funny shirt', 'christmas funny shirt', 'covid funny shirt', 'cute funny shirt sayings', 'canelo funny shirt', 'crazy funny shirt', 'charmander funny shirt', 'chanel funny shirt', 'couple funny shirt', 'funny shirt design', 'funny shirt decals', 'funny shirt design ideas', 'funny shirt designs sayings', 'funny shirt designs mens', 'funny shirt dawn', 'fun shirt designs', 'fun shirt day ideas', 'fun shirt dress', 'doug funnie shirt', 'disney princess funny shirt', 'dad funny shirt', 'dare funny shirt', 'dea funny shirt', 'disney funny shirt', 'dog mom funny shirt', 'death funny shirt', 'drinking beer funny shirt', 'donkey funny shirt', 'funny shirt etsy', 'funny short jokes', 'funny shirt svg free', 'engineer funny shirt', 'electrician funny shirt', 'electrician funny shirt design', 'eeyore funny shirt', 'exorcist funny shirt', 'electronic funny shirt']
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC