1

I'm trying to webscrape replies to comments. Below is my code to load and click the "load more comments" btn.

load_replies =driver.find_elements_by_xpath("//div[@class='thread-node-children-show']/span")
for i in (load_replies):
      WebDriverWait(driver, 10).until(EC.visibility_of(i))
      i.click()

However, this only loads a few replies, and then I receive the error "Element could not be scrolled into view". I'm not sure how to fix the error. I've tried extending the load time, but that doesn't help.

Could you please help? For reference, I'm using Python. I should also add that prior to using the for loop, my code was :

WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[@class='thread-node-children-show']/span"))).click()

However, I added the for loop because without it, that line of code would only load the first reply instead of all the replies.

Thank You.

shorttriptomars
  • 325
  • 1
  • 9
  • Could you provide us the `HTML` code? This would give us a clue in determining what the correct answer would be. A simple screenshot would help out a lot. The only thing that I could think of would be for you to use the following `xpath` in your Chrome Dev Tools ( F12 ) to see how many children display - `//div[@class='thread-node-children-show']` . Then, from there, you could write a `for-loop` that loops through each element. – Zvjezdan Veselinovic Nov 11 '20 at 01:46

1 Answers1

1

Based on reading your post, I think that you can use the following xpath in your Chome Dev Tools ( F12 ) to determine how many children elements display

//div[@class='thread-node-children-show']

Based on this result ( 1 of _ ), you could use the scrollIntoView method to navigate to the element using a for-loop.

Example Code

numOfElements = driver.find_elements(By.XPATH, "//div[@class='thread-node-children-show']").__len__()
for num in range(numOfElements):
    element = driver.find_element(By.XPATH, "//div[@class='thread-node-children-show'][{0}]".format(num + 1))
    driver.execute_script("return arguments[0].scrollIntoView();", element)
Zvjezdan Veselinovic
  • 382
  • 2
  • 7
  • thank you for taking the time to respond. I think I see the problem. If the first 4 comments have replies, the loop works and the responses load. But if the 5th comment does not have a reply, the loop breaks instead of moving on to check if comment 6 has a reply. – shorttriptomars Nov 11 '20 at 03:43
  • 1
    @shorttriptomars - Before each iteration, you could check if the element displays prior to doing anything. Meaning, if it does not display, you can use `continue` and it will skip to the next iteration of the `for-loop`. – Zvjezdan Veselinovic Nov 11 '20 at 03:56