Headless chrome with selenium, can only find ways to scroll non-headless

Question

There is a lot to find about this subject, but can't figure this out. I need to scroll to the end of the page of a (not so long) infinity scroll. I have 2 options that work with chrome non-headless but doesn't seem to work headless.

The first one which i liked the most, works beautiful and found here on SA:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('http://www.website.com')

while True:
    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    print(count)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

Second more hack job after realizing i can't get away with above in headless mode:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('https://www.website.com')

while True:

    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    actions = ActionChains(driver)
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()


    # focus_element_scroll = driver.find_elements_by_xpath('//section[@class="occasion-content"]')
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # self.driver.find_element_by_css_selector("ul.list-with-results").send_keys(Keys.ARROW_DOWN)
    print(count)
    # driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    try:  
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

So both work in chrome but don't in headless mode, i need to push them to a ubuntu vps where they need to be headless, i know of the xvfb option but i am glad i could remove that and use native chrome since the droplets don't have much memmory.

Edit: Just tried this approach with focus on an element in the footer, also works in non-headless but doesn't in headless:

ActionChains(driver).move_to_element(focus[0]).perform()

Someone got a different approach?

Edit Just want to know if it is possible to scroll with chrome in headless mode!

score 10 · Accepted Answer · answered Jan 17 '18 at 14:37

Found the answer after 2 days trying different combinations of versions of selenium, chrome and chromedriver i alsmost gave up and wanted to go with the xvfb.

Already tried to maximze the window in the chrome arguments, that didn't help. But this time i tried setting a manual window size. That helped.

    chrome_options.add_argument("window-size=1920,1080")

Posting here so that the next one wont take as long as me.

undetected Selenium · Answer 2 · 2019-06-07T10:58:14.740

2

To scroll to the end of the page of a (not so long) infinity scroll through Default Chrome Browser and Headless Chrome Browser you can use the following code block :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

options = Options()
options.add_argument("--headless")
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('http://www.website.com')

while (driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")):
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH, "//div[@itemprop='itemListElement']" )))
        # do your other actions within the Viewport
    except TimeoutException:
        break
print("Reached to the bottom of the page")

edited Jun 07 '19 at 10:58

answered Jan 15 '18 at 08:54

undetected Selenium

183,867
41
278
352

1

thanks for the awnser but this is how i have set it up, it works, bot only in non headless mode, same as the 2 other solutions i presented. Did you try this code? – userRR Jan 15 '18 at 13:51
I didn't get your comment. Doesn't my answer executes for you in **headless** mode? The code is pretty well tested at my end. – undetected Selenium Jan 15 '18 at 13:53
1

The code works well without the --headless parameter. In windows and on my ubuntu vps it doesn't work in headless mode... There takes no scrolling place. – userRR Jan 15 '18 at 14:05
@user9108711 Update the question with the error stack trace please on Windows. – undetected Selenium Jan 15 '18 at 14:15
There is no error, the WebDriverWait kicks in and the element never becomes visible because there is no scrolling. Again, i can see it clearly scrolling nice without the headless argument. The page i am trying to fetch: http:// bit.ly/2mxdV1Z – userRR Jan 15 '18 at 14:36
It also does not scrolls for me at all. No error message it joust does not scrolls https://www.oculus.com/experiences/quest/section/1888816384764129/ – sogu Jul 14 '22 at 10:24

score 0 · Answer 3 · answered Jun 05 '19 at 13:55

I just ran into this problem on windows. Using chrome 74 and i fixed the issue by having the below chromeOptions. My headless mode works again :) Thanks at DebanjanB

chromeOptions.addArguments("--headless")
chromeOptions.addArguments("--no-sandbox")
chromeOptions.addArguments("--disable-dev-shm-usage")
chromeOptions.addArguments("--window-size=1920x1080")
chromeOptions.addArguments("start-maximised")

Headless chrome with selenium, can only find ways to scroll non-headless

3 Answers3

Linked