Selenium how to scroll down to get all the link

Question

I want to export all the documents, so I need all the links.

if the mouse does not scroll down, all links will not be loaded.

quip

Need to move down a bit to gradually load

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

# Configuration information
email = "187069474@qq.com"
password = "Huangbo1019@"

driver = webdriver.Chrome()
index_url = "https://testselenium.quip.com/BCWAOAUkg1v"
driver.get(url=index_url)
driver.find_element_by_xpath('//*[@id="header-nav-collapse"]/ul/li[9]/a').click()  # click login
time.sleep(1)
driver.find_element_by_xpath('/html/body/div[2]/div[1]/div[1]/form/div/input').send_keys(email)  # input email
driver.find_element_by_xpath('//*[@id="email-submit"]').click()
time.sleep(1)
driver.find_element_by_xpath('/html/body/div/div/form/div/input[2]').send_keys(password)  # input password
driver.find_element_by_xpath('/html/body/div/div/form/button').click()
time.sleep(2)

May need a new strategy

check this post on how to do it: https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python — rs007, Sep 23 '19 at 08:52

score 0 · Answer 1 · answered Sep 23 '19 at 10:06

0

you can try directly with request library

links = browser.find_elements_by_tag_name('a')
for link in links:
    try:
        requests.get(link.get_attribute('href'))

answered Sep 23 '19 at 10:06

Boikot

304
2
9

score 0 · Answer 2 · answered Sep 23 '19 at 16:39

You can use Javascript to scroll by coordinates. If you can get the height of your page in pixels, you can call a Scroll(int xCoordinate, int yCoordinate) method & increment the scroll coordinates with each successive scroll.

You want to start by scrolling to an initial x,y coordinate on the page. As you scroll down, your y coordinate to scroll to will change. If you don't increment the y coordinate as you scroll down, you will be stuck scrolling to the same place over and over again. So you should determine your increment variable based on the pixel height of the page you are using. Depending on how many rows you want to scroll at once, your increment variable should be about equal to the pixel height of each 'row' you are scrolling past.

numberOfScrolls should be equal to the number of rows which you wish to scroll.

A method would look something like this:

public void ScrollDown(int startX, int startY, int increment, int numberOfScrolls) 
{
    // cast webdriver to javascript executor
    var executor = ((IJavaScriptExecutor) driver);

    // keep track of current y coordinate
    // since we are scrolling down, y coordinate will change with each scroll
    var currentYCoordinate = startY;

    // scroll down in a loop
    for (int i = 0; i < numberOfScrolls; i++)
    {
        // scroll down
        executor.ExecuteScript("window.scrollTo(startX, currentYCoordinate);");

        // todo: implement any FindElement methods to get the new elements which have been scrolled into view.

        // update y coordinate since we just scrolled down
        currentYCoordinate = currentYCoordinate + increment;

     }
}

Selenium how to scroll down to get all the link

2 Answers2

Linked