Web scraping Yelp, how do I retrieve the value of each individual rating?

Question

Working on a web scraping project to build my knowledge (beginner). This code is messy but I currently am to a point where I can print the rating for each review. How do I extract the rating from the bs4 object i.e. 4.0, 5,0 that is in the list, and then average them?

Output:
[<meta content="4.0" itemprop="ratingValue"/>, <meta content="5.0" itemprop="ratingValue"/>, ... ]

import mechanize
from bs4 import BeautifulSoup

def searchYelp():

    br = mechanize.Browser()
    br.set_handle_robots(False)
    br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

    response = br.open('https://www.yelp.com')
    br.select_form(nr=0)
    br.form['find_desc'] = 'Del Taco'
    br.form['find_loc'] = 'New York City'
    br.submit()

    link_list = []
    for link in br.links():
        if link.url.startswith('/biz/'):
            link_list.append(link.url)
            break

    big_list_of_ratings = []
    yelpPage = br.open(link_list[0])
    soup = BeautifulSoup(yelpPage.read(), 'html.parser')

    for review in soup.find_all('meta'):
        if review.get('itemprop') == 'ratingValue':
            big_list_of_ratings.append(review)

    print(big_list_of_ratings)


searchYelp()

Have you read the BeautifulSoup documentation? Is this not a duplicate of https://stackoverflow.com/q/2612548/11301900 ? — AMC, Jan 16 '20 at 22:28
Does this answer your question? [Extracting an attribute value with beautifulsoup](https://stackoverflow.com/questions/2612548/extracting-an-attribute-value-with-beautifulsoup) — AMC, Jan 17 '20 at 01:20

score 0 · Accepted Answer · answered Jan 16 '20 at 21:49

Instead of this

for review in soup.find_all('meta'):
        if review.get('itemprop') == 'ratingValue':
            big_list_of_ratings.append(review)

Add the attribute like this review['content']

  for review in soup.find_all('meta'):
            if review.get('itemprop') == 'ratingValue':
                big_list_of_ratings.append(review['content'])

Or I would suggest use css selector.

for review in soup.select('meta[itemprop="ratingValue"][content]'):
        big_list_of_ratings.append(review['content'])

Web scraping Yelp, how do I retrieve the value of each individual rating?

1 Answers1