0

new to python and selenium. For fun, I'm scraping a page. I have to click a first button for Comment, and then another button for All comments so I can get them all. The first click works, but not the second. I've set a hardcoded scroll, but still not working.

This is the python code I'm working on:

boton = driver.find_element_by_id('tabComments_btn')
boton.click()

wait = WebDriverWait(driver, 100)

from here on, it doesnt work (it scrolls but it says 'elem cant be scrolled into view'

driver.execute_script("window.scrollTo(0, 1300)")
botonTodos= driver.find_element_by_class_name('thread-node-children-load-all-btn')

wait = WebDriverWait(driver, 100)

botonTodos.click()

If I only click the first button, I'm able to scrape the first 10 comments, so this is working.

wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'thread-node-message')))

for elm in driver.find_elements_by_css_selector(".thread-node-message"):
    print(elm.text)

This is the part of the HTML I'm stuck in:

    <a href="#" class="thread-node-btn thread-node-children-load-next-btn">Load next 10 comments</a>
    <a href="#" class="thread-node-btn thread-node-children-load-all-btn">Load all comments</a>
    <a href="#" class="thread-node-btn thread-node-btn-post">Publicar un comentario</a>

There's a whitespace node with the tag #text between each . Any ideas welcome. Thanks.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
sophiet
  • 13
  • 7
  • Can you try to retrieve your element ```botonTodos``` before the scrolling and changing the JScript to ```driver.execute_script("arguments[0].scrollIntoView();", botonTodos)``` – Nic Laforge Mar 21 '19 at 00:48
  • Hi @Nic, thnks for answering. I wrote this `botonTodos= driver.find_element_by_class_name('thread-node-children-load-all-btn') driver.execute_script("arguments[0].scrollIntoView();", botonTodos) wait = WebDriverWait(driver, 100) botonTodos.click()` was this what you meant? Still got the same error – sophiet Mar 21 '19 at 00:58
  • `driver.execute_script("arguments[0].scrollIntoView(True);", botonTodos)` try that – KunduK Mar 21 '19 at 01:32
  • Hi @Kajal, I'm getting a 'True is not defined' now. – sophiet Mar 21 '19 at 13:24

1 Answers1

0

Here are the different options.

#first create the elements ref
load_next_btn = driver.find_element_by_css_selector(".thread-node-children-load-next-btn")
load_all_btn = driver.find_element_by_css_selector(".thread-node-children-load-all-btn")
# scroll to button you are interested (I am scrolling to load_all_btn
# Option 1
load_all_btn.location_once_scrolled_into_view

# Option 2
driver.execute_script("arguments[0].scrollIntoView();",load_all_btn)

# Option 3
btnLoctation = load_all_btn.location
driver.execute_script("window.scrollTo(" + str(btnLoctation['x']) + "," + str(btnLoctation['y']) +");")

Test Code: Check if this code is working.

url = "https://stackoverflow.com/questions/55228646/python-selenium-cant-sometimes-scroll-element-into-view/55228932?    noredirect=1#comment97192621_55228932"
driver.get(url)
element = driver.find_element_by_xpath("//a[.='Contact Us']")
element.location_once_scrolled_into_view
time.sleep(1)
driver.find_element_by_xpath("//p[.='active']").location_once_scrolled_into_view
driver.execute_script("arguments[0].scrollIntoView();",element)

time.sleep(1)

supputuri
  • 13,644
  • 2
  • 21
  • 39
  • Hi @supputuri, with the three opts I'm still getting the cannot scroll into view :/ I'm using Mozilla, could this be an issue? – sophiet Mar 21 '19 at 13:30
  • Can you please check if the test code working in Firefox. Then it's easy to figure out. – supputuri Mar 21 '19 at 13:54
  • I'm getting this error: NameError: name 'time' is not defined. I have from time import sleep, but nothing else with time. – sophiet Mar 21 '19 at 14:09
  • you have to `import time`. – supputuri Mar 21 '19 at 14:44
  • I imported time. But it wasnt working, it didnt get any error but it just doesnt scrap all the comments. I wrote this: `element = driver.find_element_by_xpath("//a[.='Load all comments']") element.location_once_scrolled_into_view print(element.location_once_scrolled_into_view) time.sleep(1) driver.execute_script("arguments[0].scrollIntoView();",element) element.click()` and on the second line in the console I get this: {'x': 0, 'y': 0}, so it's not finding the ? – sophiet Mar 21 '19 at 19:16
  • Can you please share the url , if it's ok. – supputuri Mar 21 '19 at 19:20
  • Hi, it's a p*rn site, and I dont know the community guidelines regarding sharing that kind of content on stackoverflow. – sophiet Mar 21 '19 at 19:32
  • Please `don't` share the url. – supputuri Mar 21 '19 at 19:38
  • Hahaha, thank you still. WIll try to continue with all the reccomendations. – sophiet Mar 21 '19 at 19:46