0

I want to scrape all products(40) from deals page in amazon but just get 16 one I searched a lot and found that I should use scroll but I got the same value and scroll didn't work

code

# -*- coding: utf-8 -*-
import requests
import time
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome(executable_path='C:\\Users\\Compu City\\Desktop\\chromedriver.exe')

driver.get('https://www.amazon.com/international-sales-offers/b/?ie=UTF8&node=15529609011&ref_=nav_navm_intl_deal_btn')
time.sleep(10)
res = driver.execute_script("return document.documentElement.outerHTML",'window.scrollBy(0,2000)')
soup = BeautifulSoup(res , 'lxml')
for x in soup.find_all('a',{'class':'a-size-base a-link-normal dealTitleTwoLine singleCellTitle autoHeight'}):
    for y in x.find_all('span',{'class':'a-declarative'}):
        print('\n >>>'+y.text+'\n')
driver.close()
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
colla bingo
  • 97
  • 11

1 Answers1

1

You can use the below css to get all 40 items.

div[class^='a-section a-spacing-none tallCellView gridColumn']
# below is the line to get all the products
soup.select("div[class^='a-section a-spacing-none tallCellView gridColumn']")

Screenshot:

enter image description here

supputuri
  • 13,644
  • 2
  • 21
  • 39
  • What do you mean `show`? You can use `soup.select` when you want select items based on css. – supputuri Mar 18 '20 at 16:56
  • I am not good in English (i mean the part in the screenshot which you mention for in the bottom in the pic) – colla bingo Mar 18 '20 at 16:59
  • Check [this](https://stackoverflow.com/questions/55870609/is-there-a-way-to-learn-xpath-without-using-firebug-or-xpath-as-firefox-is-not-s/55870909#55870909) post. Rather using xpath, you can try with css as I mentioned in above screenshot. – supputuri Mar 18 '20 at 17:03