0

I am trying to scrape data from a website that loads the data via javascript and also the main url does not contain page number . The url is site link . To scrape the data i am using python programming langue & selenium. But I got only first page data I couldnt get the other page like page 1 , page 2 , page 3.

I have mention the code that's i am greeting the single page data.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager


from bs4 import BeautifulSoup
from selenium import webdriver
url ='https://www.ikh.se/sv/kemikalier--smorjmedlar/fordon-kemikalier'

browser = webdriver.Chrome(ChromeDriverManager().install())

browser.get(url)
data = browser.find_element_by_id('dnsList')
dataList = data.find_elements_by_tag_name('a')
i = 1
for item in dataList:
    productName = item.find_element_by_class_name('ProdName')
    productPrice = item.find_element_by_class_name('ProdPrice')
    productCode = item.find_element_by_class_name('ProdCode')
    pName = productName.text
    pPrice = productPrice.text
    pCode = productCode.text

    print( '#SL : ' + str(i) + '  Product Code : ' + pCode + '-->' + 'ProductName : ' + pName + ' Product Price : ' + pPrice)
    i = i + 1
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Hoque MD Zahidul
  • 10,560
  • 2
  • 37
  • 40
  • 2
    you can get the whole product list from [api](https://www.ikh.se/sysNet/getProductsJSON/getProductsJSONDB.aspx?sua=2&lang=2&navid=14456615) – Shijith Dec 17 '19 at 07:54
  • i think there is no full information . – Hoque MD Zahidul Dec 17 '19 at 07:56
  • that api has all 93 products , you have to use [`requests`](https://requests.readthedocs.io/en/master/) library to the the data,in json. `Tuote` is your productName, `Hinta` is your productPrice and `Koodi` is your productCode insode the json which you get. you can get all these by directly using the api, no need to use selenium – Shijith Dec 17 '19 at 08:47
  • Another issue is that how can i call the api dynamically because the https://www.ikh.se/ has multiple sub product menu so the api url will be change also some id in the api . – Hoque MD Zahidul Dec 17 '19 at 08:57
  • another thing you can post thats as an answer so i can marked thats answer – Hoque MD Zahidul Dec 17 '19 at 08:58

2 Answers2

1

there is a 'PRODUKTLIST-PDF'

<form id="pdf-form" class="pdf-form" action="/getProductListPDF.asp?sua=2&amp;lang=2&amp;navid=14456615" method="POST" target="_blank">

where you get the navid for each menu, you can add that to navid of your api to get the product details of the menu as json, this will be faster

api (only navid changes for each menu)

https://www.ikh.se/sysNet/getProductsJSON/getProductsJSONDB.aspx?sua=2&lang=2&navid=14456615
Shijith
  • 4,602
  • 2
  • 20
  • 34
1

To extract all the Product Names, Product Prices and Product Codes you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following Locator Strategies: can use the following Locator Strategies:

  • Code Block:

    driver.get("https://www.ikh.se/sv/kemikalier--smorjmedlar/fordon-kemikalier")
product_names = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ProdName")))]
product_prices = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ProdPrice>span.currency-price")))]
product_codes = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ProdCode")))]
pages = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.dnsListPager.dnsCell.tablet-one-fifth button.dnsListPage:not(.active)")))
for i in range(len(pages)):
    pages[i].click()
    product_names.extend([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ProdName")))])
    product_prices.extend([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ProdPrice>span.currency-price")))]) 
    product_codes.extend([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ProdCode")))])
for i,j,k in zip(product_names, product_codes, product_prices):
    print("{} code is {} and price is {}".format(i,j,k))

  • Console Output:

    TVÄTTMEDEL FÖR LÅNGTRADARE KONSENTRAT 1L code is WKPR010 and price is 462.00
BILTVÄTT,KONSENTRAT 5L VIIMA code is WKPA005 and price is 218.90
BÅTTVÄTT,KONSENTRAT code is WKPV005 and price is 218.90
SILICONSTIFT 65ML code is WKSI001 and price is 39.60
VAXSHAMPOO PS-POLYGEN 10L code is HK70070 and price is 363.00
VAXSHAMPOO PS-POLYGEN 200L code is HK70071 and price is 5555.00
GLAS RENGÖRARE 500ML code is TIP731 and price is 86.90
POLISH &amp; VAX 500ML code is TIP740 and price is 126.50
COLOUR RENOVATOR code is TIP741 and price is 126.50
SCRATCH REMOWER 250ML code is TIP745 and price is 86.90
KROMI POLISH code is TIP742 and price is 126.50
SUPER SHAMPOO &amp; VAX 500ML code is TIP743 and price is 86.90
SUPERFAST VAX 500ML code is TIP744 and price is 126.50
BUMPER BLACK 500 ML code is TIP753 and price is 126.50
SHAMPOO WASH &amp; POLISH 2X500ML INK. SVAMP code is TIP756 and price is 71.50
POLERING SET BIL code is MIR100AUTO and price is 405.90
GLASTVÄTT SUPER 1L code is WKLA001 and price is 53.90
AUTOTVÄTT 2,5L code is WKS002 and price is 82.50
EFFEKTTVÄTT GEL 1L code is WKTP001 and price is 86.90
FÄLGTVÄTT GEL 1L code is WKVA001 and price is 97.90
POWERTVÄTT 4L PINELINE code is PL100004 and price is 207.90
POWERTVÄTT 10L PINELINE code is PL100010 and price is 471.90
POWERTVÄTT 200L PINELINE code is PL100200 and price is 7645.00
HEAVY DUTY 25L PINELINE code is PL133020 and price is 2035.00
RENGÖRINGSMEDELL 500ML GLAS MONTERING code is SIK037 and price is 93.50
CAR 25L PINELINE code is PL160025 and price is 2035.00
GO 25L PINELINE code is PL163025 and price is 2035.00
POWER CLEANER 0,5L PINELINE code is PL220005 and price is 108.90
FÄLGTVÄTT 0,5L PINELINE code is PL350005 and price is 93.50
INSEKT ELIMINERING 0,5L PINELINE code is PL373005 and price is 82.50
GLASS CLEANER 0,5L PINELINE code is PL444005 and price is 75.90
SKUM TVÄTT 25L PINELINE AKTIV SKUM code is PL500025 and price is 1210.00
POWERTVÄTT 25L PINELINE code is PL100025 and price is 1089.00
POWERTVÄTT SUPER 25L PINELINE code is PL130025 and price is 1419.00
CAR+GO 25L PINELINE code is PL200025 and price is 748.00
FÄLGTVÄTT 4L PINELINE code is PL350004 and price is 438.90
SKUM TVÄTT 4L PINELINE AKTIV SKUM code is PL500004 and price is 438.90
KRAFTAVFETTNING 1L MED PUMP code is HAG001 and price is 82.50
KRAFTAVFETTNING 4L code is HAG002 and price is 328.90
FÖRTVÄTT PLUS 1L PUMP code is HAG003 and price is 82.50
FÖRTVÄTT PLUS 5L code is HAG004 and price is 328.90
SUPER TVÄTT 500ML code is HAG005 and price is 64.90
VAX TVÄTT 500ML code is HAG006 and price is 64.90
SKUM TVÄTT 500ML code is HAG007 and price is 64.90
SNABB TVÄTT 1L PUMP code is HAG008 and price is 109.45
INSEKTSBORTTAGARE 0,5 L PUMP code is HAG010 and price is 75.90
FÄLGRENGÖRING 500ML PUMP code is HAG011 and price is 64.90
FÖNSTERPUTS 500ML PUMP code is HAG012 and price is 53.90
INTRRIÖRRENGÖRING 500ML PUMP code is HAG013 and price is 64.90
TEXTIL RENGÖRING 500ML PUMP code is HAG014 and price is 64.90
LÄDER RENGÖRING 300ML code is HAG015 and price is 75.90
ROSTÄTARE 300ML code is HAG023 and price is 108.90
ROSTOMVANDLARE 150ML code is HAG024 and price is 148.50
UNDERREDSMASSA 1KG SVART code is HAG026 and price is 126.50
BROMSVÄTSKA 0,5L DOT 4 code is WKJ0005 and price is 43.45
BROMSVÄTSKA DOT 4 4L code is WKJ004 and price is 284.90
LHM PLUS SPECIALOLJA 1L code is WKLHM1 and price is 71.50
BASF HYDRAULAN 404 BROMSVÄTSKA DOT 4 1L code is BASF404-1 and price is 108.90
KYLARVÄTSKA 1L 100% code is WK001100 and price is 44.00
KYLARVÄTSKA 3L 100% code is WK003100 and price is 124.30
KYLARVÄTSKA 5L 100% code is WK005100 and price is 192.50
KYLARVÄTSKA 100% 10L code is WK010100 and price is 328.90
KYLARVÄTSKA 200L 100% code is WK200100 and price is 6380.00
KYLARVÄTSKA 100% SUPER HD 5L code is WK005100P and price is 229.90
KYLARVÄTSKA 100% SUPER HD 10L code is WK010100P and price is 423.50
REPARATIONS MEDEL FÖR KYLSYSTEM code is TSL001 and price is 68.20
KENT RADIATOR LEAK STOP 250ML code is KENT86182 and price is 438.90
KYLARVÄTSKA 100% LL OAT 10L code is WK010100K and price is 438.90
BASF KYLARVÄTSKA G30 1,5L KONCENTRERAD code is GLYSG30-1 and price is 143.00
BASF KYLARVÄTSKA G40 1,5L KONCENTRERAD code is GLYSG40-1 and price is 148.50
BASF GLYSANTIN G48 1,5L KONCENTRERAD code is GLYSG48-1 and price is 148.50
BASF KYLÄRVÄTSKA G64 5L FÄRDIGBLANDAD 50% code is GLYSG64-5R and price is 407.00
POLERINGSPASTA SATS 2ST EFTERBEHANDLING code is TPG35937 and price is 76.45
POLERINGSPASTA ASSORTMENT 2ST code is TPG35938 and price is 76.45
POLARSHINE LIQUID NANO WAX 1L code is MIR1491 and price is 363.00
POLARSHINE NANO ANTISTAT WAX UF3 1L code is MIR1492 and price is 330.00
POLARSHINE 35 1L code is MIR1494 and price is 352.00
POLARSHINE 10 1L code is MIR1561 and price is 330.00
POLERING OCH SLIPNING SERIE FÖR STRÅLKASTARE code is OB039 and price is 181.50
POLARSHINE 35 250ML code is MIR1580 and price is 154.00
POLARSHINE 10 250ML code is MIR1581 and price is 162.80
POLERING SET BÅT code is MIR100VENE and price is 319.00
POLARSHINE WAX LIQUID NANO 500ML code is MIR1622 and price is 231.00
POLARSHINE 20 250ML code is MIR1699 and price is 242.00
POLARSHINE 20 1L code is MIR1700 and price is 495.00
POLISH GROV RUBBING 300ML code is HAG016 and price is 75.90
POLISH FIN LACKRENGÖRING 300ML code is HAG017 and price is 75.90
VAXPOLISH 300ML code is HAG018 and price is 75.90
GLANSVAX 300ML code is HAG019 and price is 75.90
LACKFÖRSEGLING 300ML code is HAG020 and price is 97.90
DÄCK OCH FÄLGGLANS 500ML PUMP code is HAG021 and price is 97.90
POLARSHINE   5 250ML code is MIR1709 and price is 187.00
POLARSHINE   5 1L code is MIR1710 and price is 495.00
SOMMARSPOLARVÄSTKA 4L code is WKKE004 and price is 30.25
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352