Python 3 Web Scrape & Beautiful Soup Tag Attribute

Question

I am practicing on Beautiful Soup and am after a products price, description and item number. The first 2 are text and are easy to get. The third is an attribute of the tag data-trade-price as seen below:-

<div class="price-group display-metro has-promo-price  medium ng-scope" ng-class="{'has-trade-price': ShowTrade}" data-trade-price="221043">

I am after the numbers such as 221043 which is loaded in by the page. IE - all 24 item numbers matching all 24 products

My code is:-

import requests  
r = requests.get('http://www.supercheapauto.com.au/store/car-care/wash-wax-polish/1021762?page=1&pageSize=24&sort=-ProductSummaryPurchasesWeighted%2C-ProductSummaryPurchases')

from bs4 import BeautifulSoup  
soup = BeautifulSoup(r.text, 'lxml')  
results = soup.find_all('div', class_='details')

for result in results: 
    try:
        SKU = result.select_one("data-trade-price")
    except AttributeError: SKU = "N/A"

    DESC = result.find('div', class_='title').text.strip().upper()
    PRICE = result.find('span', class_='currency').text.strip().upper()

    print(SKU,'\t', DESC,'\t', PRICE)

What is the syntax to get the item number from the soup?

Sorry - I am after the syntax that can iterate through the page of 24 products and recover the 24 different item numbers. The example given was to show the part of the attribute value that I was after. I ran the given answer and it works. I am unsure of how to integrate into the code given as the variations I use do not. Any suggestions.

Possible duplicate of [Python Beautifulsoup Getting Attribute Value](https://stackoverflow.com/questions/50533363/python-beautifulsoup-getting-attribute-value) — Aankhen, Jul 11 '18 at 07:07

score 0 · Accepted Answer · answered Jul 11 '18 at 06:13

0

You can access the attribute just like a dictionary.

Ex:

from bs4 import BeautifulSoup
s = """<div class="price-group display-metro has-promo-price medium ng-scope" ng-class="{'has-trade-price': ShowTrade}" data-trade-price="221043"<\div>"""
soup = BeautifulSoup(s, "html.parser")
print( soup.find("div", class_="price-group display-metro has-promo-price medium ng-scope").attrs["data-trade-price"] )

or

print( soup.find("div", class_="price-group display-metro has-promo-price medium ng-scope")["data-trade-price"] )

Output:

answered Jul 11 '18 at 06:13

Rakesh

81,458
17
76
113

I don't think that'll work with requests when that value is likely generated by Angular – OneCricketeer Jul 11 '18 at 06:15
It looks like the page at the URL given has `data-trade-price` in the source, so it should be fine. – Aankhen Jul 11 '18 at 07:06
Sorry - I am after the syntax that can iterate through the page of 24 products and recover the 24 different item numbers. The example given was to show the part of the attribute value that I was after. I ran the given answer and it works. I am unsure of how to integrate into the code given as the variations I use do not. Any suggestions. – Tony H Jul 11 '18 at 11:16
Thanks Rakesh - re-checked the HTML output and noticed that not all classes were the same based on different prices. Shortened the find to (class_='price-group'.attrs['data-trade-price]) and it pulled all the data I was after. – Tony H Jul 14 '18 at 00:21

Python 3 Web Scrape & Beautiful Soup Tag Attribute

1 Answers1