Cannot Scroll Webpage by Selenium Webdriver or even Javascript Executor

Question

`from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import time

driver = webdriver.Chrome('chromedriver.exe')

driver.get('https://iremedy.com/search?query=Vital%20Signs%20Monitors')

time.sleep(5)

element = driver.find_element(By.CLASS_NAME, 'body').send_keys(Keys.END)`

I have tried various methods but none are working. Kindly help me.

Does this answer your question? [How can I scroll a web page using selenium webdriver in python?](https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python) — Carapace, Sep 13 '22 at 13:59
What error do you get? Is the page opens or do you get an exception — Tal Angel, Sep 13 '22 at 14:04
@Carapace I Checked all the mentioned methods but none worked unfortunately. — Muhammad Talha Zeb Khan, Sep 13 '22 at 14:09

Barry the Platipus · Accepted Answer · 2022-09-13T14:13:55.473

2

This is one way of scrolling that page and loading the items:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import pandas as pd
import time as t

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')

chrome_options.add_argument("window-size=1280,720")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
url = 'https://iremedy.com/search?query=Vital%20Signs%20Monitors'
browser.get(url)
items_list = []
while True:
    elements_on_page = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '[class^="card"]')))
    print(len(elements_on_page), 'total items found')
    if len(elements_on_page) > 100:
        print('more than 100 found, stopping')
        break
    footer = wait.until (EC.presence_of_element_located((By.CSS_SELECTOR, 'footer[id="footer"]')))
    footer.location_once_scrolled_into_view
    t.sleep(2)
for el in elements_on_page:
    title = el.find_element(By.CSS_SELECTOR, 'h3[class="title"]')
    price = el.find_element(By.CSS_SELECTOR, 'div[class="price"]')
    items_list.append((title.text.strip(), price.text.strip()))
df = pd.DataFrame(items_list, columns = ['Item', 'Price'])
print(df)

The result printed in terminal will be:

10 total items found
20 total items found
20 total items found
30 total items found
30 total items found
40 total items found
50 total items found
60 total items found
70 total items found
80 total items found
90 total items found
100 total items found
110 total items found
more than 100 found, stopping
Item    Price
0   Edan M3A Vital Signs Monitors   $2,714.95
1   M3 Vital Signs Monitors by Edan Instruments $2,476.95
2   Vital Signs Patient Monitors - Touch Screen $2,015.95
3   RVS-100 Advanced Vital Signs Monitors by Riester    $362.95
4   Edan iM80 Vital Signs Patient Monitors  $5,291.95
... ... ...
105 Patient Monitor Connex® Vital Signs Monitoring...   $10,571.95
106 Patient Monitor Connex® Spot Check and Vital S...   $5,089.95
107 Patient Monitor Connex® Spot Check and Vital ...    $5,964.95
108 Patient Monitor X Series® Vital Signs Monitori...   $48,978.95
109 Patient Monitor Connex® Spot Check and Vital S...   $4,391.95
110 rows × 2 columns

I'm breaking the loop once I reach 100, you can go higher.. Important to note there is another way to obtain that data as well, by scraping the GraphQL endpoint where the data is being pulled from. Nonetheless, this is how you do it with Selenium. For documentation, please see https://www.selenium.dev/documentation/

edited Sep 13 '22 at 14:13

answered Sep 13 '22 at 14:08

Barry the Platipus

9,594
2
6
30

Thanks a lot! This worked. Made my day :). Can you kindly share some link related to GraphQL endpoint scrapping. GrapghQL seems a better viable option. I am not aware of this method so kindly point point me from where to start. Thanks a lot. – Muhammad Talha Zeb Khan Sep 13 '22 at 14:13
I edited my response, to make the retrieved data more .. decent. Regarding GraphQL.. that will require a long, complex header, and to be honest with you, a better solution (less complex) in this scenario remains Selenium. – Barry the Platipus Sep 13 '22 at 14:15
Got it ✅, this surely seems easy for me to understand, still in the learning phase. – Muhammad Talha Zeb Khan Sep 13 '22 at 14:21
@BarrythePlatipus Do you know why `element.location_once_scrolled_into_view` worked here while other ways not? Anyway +1 for the good catch! – Prophet Sep 13 '22 at 14:22
@Prophet `location_once_scrolled_into_view` will scroll the element into view and determine its location on screen (you can print it and it will be a dict of xy coords). It's a way to scroll the unscrollable (and poetic!). It appears to be under-documented, but I use it since times immemorial, since I stumbled upon it probably in a SOF question. Maybe we could use some reference to it in Community Wiki? – Barry the Platipus Sep 13 '22 at 14:25
This is what I asked: why this working why other approaches not? After your solution I saw an old question about scrolling with the second, less upvoted answer there showing this way and giving explanations. My question, again, is why this is working while other approaches not? – Prophet Sep 13 '22 at 14:28
Maybe because other approaches are trying to interact with the page visible body, which is covered with all sorts of layers (divs etc), so it's difficult to find the actual element which allows interaction - clicking or sending Keys.END, etc. – Barry the Platipus Sep 13 '22 at 14:30
1

@BarrythePlatipus Just one last thing, I hope I'm not asking a lot. Each card has a three dot submenu to access quick look, Is it possible to open that of each card and scrap the description part as well. Initially I thought of going to each page and then going back, but thought it would be an additional overhead. Want to know if its possible just by quick look menu. Some reference code would be appreciated. – Muhammad Talha Zeb Khan Sep 13 '22 at 14:33
Dear @MuhammadTalhaZebKhan, this is a new question: please post it as a new question, and me or someone will surely look into it and try to give you a solution. – Barry the Platipus Sep 13 '22 at 14:34
You're welcome @Prophet, and make no mistake: it's still me who learned more from your answers, than the other way around :) – Barry the Platipus Sep 13 '22 at 14:35
Acknowledged ✅✅✅✅ – Muhammad Talha Zeb Khan Sep 13 '22 at 14:35
@BarrythePlatipus This is exactly what this platform for: to learn one from another :) – Prophet Sep 13 '22 at 14:37
@BarrythePlatipus Don't want to disturb you, but I have posted it: https://stackoverflow.com/questions/73705734/can-each-link-of-card-be-opened-in-selenium-while-scrapping-infinte-scroll – Muhammad Talha Zeb Khan Sep 13 '22 at 15:40

Cannot Scroll Webpage by Selenium Webdriver or even Javascript Executor

1 Answers1