NameError: name is not defined web scraper

Question

Hi I'm trying to build a simple webscraper of ESG data for S&P500 companies but for some reason I keep getting NameError:

name 'web_scraper' is not defined

Tried a few things now but cannot get it to work. I'm a beginner so must be overlooking something. Thanks in advance

from bs4 import BeautifulSoup
import pandas as pd
import requests
import ipywidgets as widgets
from ipywidgets import interact

def get_tickers():
 # wiki_page = requests.get()
  sp_data = pd.read_html('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
  ticker_df = sp_data[0]
  ticker_options = ticker_df['Symbol']
  return ticker_options

  def level(x):
    if x == 0.0:
        return 'No Controversy'
    elif x == 1.0:
        return 'Little Controversy'
    elif x == 2.0: 
        return 'Moderate Controversy'
    elif x == 3.0: 
        return 'Relatively High Controversy'
    else:
        return 'Little Controversy'

    def web_scraper(ticker):
        elements = []
        web_data = requests.get('https://finance.yahoo.com/quote/'+ticker+'+/sustainability?p='+ticker).text
        soup = BeautifulSoup(web_data, 'html.parser')
        esg_score = soup.find('div', {'class':'Fz(36px) Fw(600) D(ib) Mend(5px)'})
        datapoint = esg_score.text
        controversy_score = soup.find('div', {'class':'D(ib) Fz(36px) Fq(500)'})
        controversy_datapoint = controversy_score.text
        scores = soup.find_all('div', {'class': 'D(ib) Fz(23px) smartphone_Fz(22px) Fq(600)'})
        for score in scores:
            elements.append(score.text)

    df = pd.DataFrame({'Total ESG Score': datapoint,
                   'Environment Score': elements[0],
                   'Social Score': elements[0],
                  'Governance Score': elements[2],
                 'Controversy Score': controversy_datapoint},
                  index = [ticker])
    df = df.astype('float')
    df['Controversy Assessment'] = df.apply(lambda x: level(x['Controversy Score']), axis=1)
    return df

@interact
def get_esg(ticker=widgets.Dropdown(description='Ticker',
                                      options=get_tickers(),
                                    value = 'F')):
  esg_data = web_scraper(ticker)
  return esg_data

score 0 · Answer 1 · answered Nov 18 '20 at 22:13

0

Because of your indentation, def web_scraper is defined in def level, which is defined inside def get_tickers. So, if you want to be able to access web_scraper from anywhere, you have to define it in global scope, not just within the scope of another function (which is itself within yet another function).

answered Nov 18 '20 at 22:13

Random Davis

6,662
4
14
24

Thank you Random Davis. It did fix the issue I was having. When I run the code right now I get `Indexerror: list index out of range` for the elements in the df. Any chance you see why that is the case? – toost Nov 19 '20 at 14:29
@toost since that's a separate question, you can just make a new post, it's fine. You're meant to ask only one question per post. – Random Davis Nov 19 '20 at 18:54

NameError: name is not defined web scraper

1 Answers1