0

Selenium is really important here. So, I want to create a program that can help me scrap stuff from google like the snippets etc while also giving me the ability to automate the browser for some other tasks. And here's what I've done.

from selenium import webdriver as webd
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webd.Firefox()

driver.get('https://google.com/')
driver.find_element(By.NAME, 'q').send_keys('cars')
driver.find_element(By.NAME, 'q').send_keys(Keys.ENTER)
ques = driver.find_element(By.CLASS_NAME, "ULSxyf")
print(ques)

time.sleep(8)
driver.close()

Although the first few lines works fine. I'm unable to open the People also ask section with selenium no matter what I do. I've used the class name of the object by inspecting it or used the id etc etc. And after searching for awhile I haven't really found anything much that would help this specific case scenario. I need to know how exactly to do this or why my method isn't working, if anybody has any idea. I'd be glad if you let me know. Thanks!

I'm a total beginner in selenium so if you can't give me a straight answer but feel a article or tutorial would be better, that would help as well.

EDIT: I want to open the questions from People also ask section and extract the answers from in there along with the questions themselves.

Xanta_Kross
  • 35
  • 1
  • 8

1 Answers1

0

To extract the texts from the questions under People also ask column you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies:

  • Code Block:

    driver.get("https://google.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.NAME, "q"))).send_keys("cars" + Keys.RETURN)
print([my_elem.get_attribute("data-q") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[text()='People also ask']//following::div[@data-q]")))])

  • Console Output:

    ['Which is the most popular car?', 'What are top 10 cars?', 'Which type of car is best?', 'Which car is very cheapest?']

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thank you for taking the time to answer the question. But I was mostly asking how I could get the text from "within" the people also ask column. Like 1) google search cars 2) click on the first "people also ask" 3) scrap the text that is now visible. I apologize I didn't make that clear in my question here I'll edit it. – Xanta_Kross Jun 19 '22 at 00:21