0

Im trying to scrape PH news website https://www.philstar.com/ but its search results is a pop-up so I have to manually click search and everything using the driver.

search_button = driver.find_element(By.TAG_NAME, "img#search_toggle")
search_button.click()
        
search_input = driver.find_element(By.TAG_NAME,"input#zoomd")
search_input.send_keys(KEYWORD)
search_input.send_keys(Keys.RETURN)

After searching using any keyword there is a button that says "More From TOP SEARCHES" that I want to click to load more news but I cant seem to find it with the driver

wait = WebDriverWait(driver, 10)

XPATH = f'//div[@class="zoomd-widget-content-wrapper"]/zoomd-search-results[3]/section/button'
        
button = wait.until(EC.element_located_to_be_selected((By.XPATH,XPATH)))

button.click()

it seems to be part of a zoomd widget. any idea how to scrape from this?

Hurly119
  • 1

1 Answers1

0

Can you try to pass the full xpath and see if it works? For the zoomd input, this is the path

/html/body/zoomd-widget-root/zd-widget/div/div[2]/div/zoomd-search-results[3]/section/button

The button is indeed being loaded dynamically, so try to add an explicit wait above the click call to make sure that it's loaded.

AzyCrw4282
  • 7,222
  • 5
  • 19
  • 35
  • I just tried this but still the same problem. The pop up seems to be a separate html element from the main page and for some reason the driver can't detect it. I tried looping through all iframes to find the button but it still doesnt exist. I also checked windowHandles but there seems to be only one for the main page. – Hurly119 Mar 13 '21 at 03:18
  • @Hurly119 Thanks for the info. It seems like it dynamically defines a custom HTML element for the data displayed on the page. In order to solve this, check this solution [here](https://stackoverflow.com/questions/55876620/how-to-click-custom-html-tag-with-selenium) , or any related solutions that you can find. You can find more on custom HTML elements [here](https://stackoverflow.com/questions/5682943/how-to-create-custom-tags-for-html). This should solve your problem. Do update otherwise – AzyCrw4282 Mar 14 '21 at 14:20