Extracting string data from a html source

Question

from bs4 import BeautifulSoup
import urllib.request

page = urllib.request.urlopen('https://www.applied.com/categories/bearings/accessories/adapter-sleeves/c/1580?q=%3Arelevance&page=1')
html = page.read()
soup = BeautifulSoup(html)

items = soup.find_all(class_= 'product product--list ')

for i in items[0:1]:
    product_name = i.find(class_="product__name").a.string.strip()
    print(product_name)
    product_url = i.find(class_="product__name").a['href']
    print(product_url)
    price = i.find(itemprop="price").string
    print(price)

Using the above code I tried to get the price for each product in that page. But when i tried, the output for price variable is showing as none.

When I inspect the html source for the price in a browser it is showing the price as a normal text as how I got for product_name variable.

Can someone guide me on how to get the price for the products in that page.

I've tested your code: the product price is initially empty in page content: ` `. Voting to close — RomanPerekhrest, Sep 18 '17 at 07:27
The price is loaded by Javascript. Try this: https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python — SajidSalim, Sep 18 '17 at 07:38

score 1 · Answer 1 · answered Sep 18 '17 at 07:39

Price is loaded by Ajax(https://www.applied.com/getprices) after page is loaded that's why it is not in HTML.

Use https://www.applied.com/getprices to get the price of an item You have to send post request with following params for getting the price of the product.

{
  "productCodes": "100731658",
  "page": "PLP",
  "productCode": "100731658",
  "CSRFToken": "172c7073-742f-4d7d-9c97-358e0d9e631e"
}

Extracting string data from a html source

1 Answers1