Url is https://donstroy.com/full-search. On that page, there is parent <div class="content"> and 21 <div class="item"> inside it. When human hovers mouse over <div class="content"> and scrolls down, multiple new <div class="item"> elements dynamically appear.
I'm trying to acheive this result (appearing of new elements) by selenium but all i can get is only first 21 <div class="item">.
I tried different methods to scrolldown in selenium but none of them resulted in appearing new <div class="item"> elements. Below is my code. The first method with scrollIntoView is working fine when i run it in pure javascript in Element Inspector's console (it causes appearing of new elements) but still not working via selenium.
import selenium.webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.action_chains import ActionChains
def parse_objs():
return driver.find_elements_by_xpath('//div[@class="item"]')
options = Options()
options.headless = True
driver = selenium.webdriver.Firefox(options=options)
driver.set_page_load_timeout(300) # yes, it may take up to multiple minutes to load the page so may need to wait long time, it's normal
driver.get('https://donstroy.com/full-search')
objs = parse_objs()
# Amount of object on initial page load
print('Initial load : %s' % len(objs)) # prints 21 as expected
# Method 1
driver.execute_script('arguments[0].scrollIntoView();', objs[-1])
objs = parse_objs()
print('Method 1 : %s' % len(objs)) # expected to increase but still prints 21
#The thing is that when running the same method in pure javascript in Element Inspector's console, it's working fine:
#var objs=document.getElementsByClassName('item');
#objs[20].scrollIntoView();
#after that, amount of objs (objs.length) increases to 41 so it should work in selenium but it does not!
# Method 2
actions = ActionChains(driver)
actions.move_to_element(objs[-1]).perform()
objs = parse_objs()
print('Method 2 : %s' % len(objs)) # expected to increase but still prints 21
# Method 3
objs_container=driver.find_element_by_xpath('//div[@class="content"]')
driver.execute_script('arguments[0].scrollTop = 300;', objs_container)
print('ScrollTop=%s' % driver.execute_script('return arguments[0].scrollTop', objs_container)) # always returns 0 no matter what value i try to apply for scrollTop in above command
objs = parse_objs()
print('Method 3 : %s' % len(objs)) # expected to increase but still prints 21
driver.quit()
