0

Im trying to scrape articles from multiple pages from https://edm.com/news using Selenium with chromedriver and ran into multiple errors while trying to hit the "See More" button. Any ideas of what can i try?

I have tried working with ActionChains.move_to_element(..).click.perform() also tried multiple time.sleep calls or WebDriverWait.until... nothing seems to do the trick.

start_url = "https://edm.com/news"
browser = webdriver.Chrome(executable_path='chromedriver.exe',      options=self.option)
browser.get(self.start_url)
# Wait max 10 secs for page to load
timeout = 10
WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.XPATH,'//*[@id="lyra-wrapper"]/div/div[3]/section/'
                                     'div[2]/section[2]/section/div/button')))
time.sleep(2)
button = browser.find_elements(By.XPATH, '//*[@id="lyra-wrapper"]/div/div[3]/section/'
                                         'div[2]/section[2]/section/div/button')[0]
button.click()
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (508, 4270)
  (Session info: chrome=74.0.3729.169)
  (Driver info: chromedriver=2.42.591088 (7b2b2dca23cca0862f674758c9a3933e685c27d5),platform=Windows NT 10.0.17763 x86_64)```

eyllanesc
  • 235,170
  • 19
  • 170
  • 241
dzz
  • 33
  • 7

2 Answers2

1

try finding the element by the text "See More"

start_url = "https://edm.com/news"
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
browser.get(start_url)

# Wait max 10 secs for page to load
wait = WebDriverWait(browser, 10)
time.sleep(2)                    

close_advert = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="lyra-wrapper"]/div/div[4]/phoenix-ad[2]/div/div[2]')))
close_advert.click()

see_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[contains(.,"See More")]'))) 

time.sleep(2)

try:
    see_more.click()
except:
    time.sleep(3)
    see_more.click()
chitown88
  • 27,527
  • 4
  • 30
  • 59
  • Hey @chitown88 first of all i really appreciate the quick reply so thanks for that in advance. Unfortunately i still run on the same "Element is not clickable at ..." error after making the changes you recommended. – dzz May 27 '19 at 10:57
  • odd. it's working on my end. might need to put a time delay in there. I'll ad it above – chitown88 May 27 '19 at 10:58
  • oh you know what. there s a pop up that blocks it which is why its saying there's no element that is clickable. You'll have to have selenium close the advert that shows up, when it shows up – chitown88 May 27 '19 at 11:04
  • Well we have some progress! Now i managed to hit it some times then got another error( other element would get the click, element not available at (x,y) ) but i think i can fix that with some delay. Cant believe it was the cookies pop up....Thanks for your help my friend ! – dzz May 27 '19 at 11:14
  • ya. for some reason too, it wasn't recognising it until it scrolled down. So it's sort of a "hack" solution, but it's working for me now. – chitown88 May 27 '19 at 11:15
  • Works perfectly also for me ! A rookie question since i've been working on Xpath and Web Scraping only for 2 months now.. Inside the contains function brackets does the . refers to text? Shouldnt it be contains(text(),string)? – dzz May 27 '19 at 11:20
  • good question. I'm still learning about the xpath stuff as well. There's a detailed explanation [here](https://stackoverflow.com/questions/3655549/xpath-containstext-some-string-doesnt-work-when-used-with-node-with-more), which is I'm still trying to grasp too – chitown88 May 27 '19 at 11:33
  • @dzz if the solution was what you needed, be sure to accept the solution above by clicking on the "check" – chitown88 May 30 '19 at 14:27
0
<button class="m-component-footer--loader m-button" onclick="return phoenixTrackClickEvent(this, event);" phx-track-id="load more">
    <div class="m-component-footer--text m-component-footer--loading">Loading…</div>
    <div class="m-component-footer--text m-component-footer--button-text">See More</div>
</button>

If you look closely, button tag has 2 div in it. So you need to use xpath for See More div which is //*[@id="lyra-wrapper"]/div/div[3]/section/div[2]/section[2]/section/div/button/div[2]. Then use click() on it.

  • Hey Saurabh thanks for the quick reply, appreciate it! The problem is it detects the See More div/button and still ends up on the not clickable at (x,y) error. Really cant get my head over why this is happening. – dzz May 27 '19 at 10:59