0

I had a look at all the similar questions on SO but couldn't manage to solve my problem. Selenium does not seem to locate a button I am trying to click on to load more the page before scraping.

In the picture below is the button I am trying to click on to load more the page before doing some scraping (class name of button : btn btn-default) enter image description here

What I have done: I used Selenium's webdriver. I tried waiting for at least 5 seconds for the page to load using sleep. And lastly using the click() function. I have inspired myself from this tutorial (section : Using Selenium)

urlpage = 'theurl'

My code is the following :

 driver = webdriver.Firefox()
driver.get(urlpage)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
time.sleep(5)
button = driver.find_element_by_class_name("btn btn-default")
button.click()

I would appreciate if someone had an idea of what I am doing wrong :)

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
colla
  • 717
  • 1
  • 10
  • 22

1 Answers1

1

The problem probably comes from the fact that you are trying to find a class name by giving 2 class names ("btn" and "btn-default"). When I changed the line to

button = driver.find_element_by_class_name("btn-default")

it seemed to work just fine. Be aware that there are few elements with the class "btn-default" and it will return only the first one. You can use the function

find_elements_by_class_name(class_name)

to find all the elements with this class (will return a list object).

Netanel
  • 477
  • 2
  • 11