Elements get disappeared when I scroll down in selenium

Asked Aug 31 '22 at 23:10

Active Nov 29 '22 at 21:57

Viewed 48 times

I'm working on a web scraping project where I'm scraping a product related to a specific niche and this is the URL where I scrap data from : https://www.aliexpress.com/af/category/200010058.html?categoryBrowse=y&origin=n&CatId=200010058&spm=a2g0o.home.108.18.650c2145CobYJm&catName=backpacks

I used selenium as a middleware to render javascript and this is the code for that :

def process_request(self, request, spider):
    self.driver.get(request.url)
    WebDriverWait(self.driver, 10).until(
        EC.element_to_be_clickable((By.CLASS_NAME, '_3t7zg'))
    )
    # self.driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
    time.sleep(10)
  
   links = self.driver.find_elements("xpath", '//div[1]/div/div[2]/div[2]/div/div[2]/a[@href]')

The problem is that on the webpage there are 60 products, when I look at the number of products I have scraped I find 11. I don't know what the problem is.

edited Nov 29 '22 at 21:57

TylerH

20,799
66
75
101

asked Aug 31 '22 at 23:10

Azzedine

It looks like lazy loading is implemented there's an infinite page scroll code you can use for that. https://stackoverflow.com/questions/22702277/crawl-site-that-has-infinite-scrolling-using-python – Arundeep Chohan Aug 31 '22 at 23:34
Can you confirm the url? – Barry the Platipus Sep 01 '22 at 11:51
@platipus_on_fire the URL is a web page for AliExpress products – Azzedine Sep 01 '22 at 12:12

Elements get disappeared when I scroll down in selenium

0 Answers0