Selenium Python: How to Scroll Down in a Pop Up window

Question

I am working on a Linkedin web scraping project. I am trying to get the list of companies that interest someone (notice I am not using the API). It is a dynamic website, so I would need to scroll down while scraping the names of the companies. I know how to do this in the MAIN window, but since Interest are a pop-up window this solution to scroll does not work. My code so far was:

from selenium.webdriver.common.keys import Keys
bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
bar.send_keys(Keys.END)

Since it didn't work, I also tried:

bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', bar)

The problem is that I am not acting on the pop up window but in the main one so it has not the desired effect.

enter image description here

Find the "scroll down" arrow css and click it until you find the element? That's how a human would do it. — BoboDarph, Aug 30 '17 at 13:15
Yes, I agree with JeffC. I added my code so far to make it more complete. — G. Macia, Aug 30 '17 at 21:06

score 9 · Answer 1 · answered Aug 30 '17 at 13:22

9

You can try to find element inside popup (the one that can be focused), for example some anchor:

element_inside_popup = driver.find_element_by_xpath('//div[@class="entity-list-wrapper ember-view"]//a')

and then use below code to scroll popup down:

from selenium.webdriver.common.keys import Keys

element_inside_popup.send_keys(Keys.END)

answered Aug 30 '17 at 13:22

Andersson

51,635
17
77
129

Just a clarification: what does '//a' exactly do? I realized that without it, the code cannot focus the element. How would the driver know when the element is over so that it does not need to scroll down any more? – G. Macia Aug 30 '17 at 21:15
`//a` means to return you the any first link on popup. Do you want to scroll to the very bottom of popup or to specific element? – Andersson Aug 30 '17 at 21:17
Thanks! I understand the '//a' magic now. I want to get the list of all companies and I realized this is only displayed fully if I get to the end of the element. – G. Macia Aug 30 '17 at 21:23
Oh. I see. You need to make XHR while complete list not received? Try `element_inside_popup.send_keys(Keys.END)` in a while loop. You should check the count of companies and if on some iteration there is no new companies- break the loop – Andersson Aug 30 '17 at 21:29
I get this error selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of stale; either the element is no longer attached to the DOM or the document has been refreshed . any way to get arround it without time.sleep a bunch? – Tintinabulator Zea Oct 16 '17 at 15:15
I need to check your code. Can you create new ticket and describe your issue there? – Andersson Oct 16 '17 at 15:31
@Andersson Can you share the code with while loop and break statement to scroll the popup till end? – Priyansh gupta Jun 01 '21 at 12:02

score 2 · Answer 2 · answered Feb 22 '22 at 15:42

2

first I tried with this script but it doesn't work:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    scrollable_popup.send_keys(Keys.END)
    time.sleep(2)

and then I used this script and it works fine with me:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_popup)
    time.sleep(2)

answered Feb 22 '22 at 15:42

Dhifaoui Bilel

41
2

Both is not working – Aalexander Sep 19 '22 at 17:22
The second one is working fine for me .. verify the XPATH of the popup – Dhifaoui Bilel Sep 21 '22 at 15:25

Selenium Python: How to Scroll Down in a Pop Up window

2 Answers2

Linked