0

Forgive me for my ignorance, I am just starting to learn python/webscraping. I'm trying to make a function which scrapes job titles from Indeed.com and it seems like I'm not able to see the full "case" for an h2 tag. The code that works is below:

def get_url(position,location):
    template='https://www.indeed.com/jobs?q={}&l={}'
    url = template.format(position, location)
    return url

url =get_url('data analyst', 'springfield ma')
response= requests.get(url)
# print(response.status_code)
soup = BeautifulSoup(response.text,'lxml')  

cards = soup.find_all('div', class_="job_seen_beacon")
# len(cards)

for i in cards:
    j=i.find('tbody')
    a= j.find('tr')
    
    for n in a.find_all('h2', class_='jobTitle jobTitle-color-purple jobTitle-newJob'):
        job_title = n.find_all('span')[1].get_text()               #if you dont [1] you get the first <span> which is "new"
        print(job_title)

The issues lies in the second iteration block where I:

find_all('h2', class_='jobTitle jobTitle-color-purple jobTitle-newJob')

This only works because I looked up a youtube tutorial on webscraping Indeed and this was the code used. When I inspect element and look at 'h2' the class is: class_="jobTitle jobTitle-newJob"

I've tried using the above class and the code returns nothing when I print. I know I'm missing something so please let me know if you can point me in the right direction. Thank you for your time.

edit: picture of what I'm seeing enter image description here

AceCharm101
  • 17
  • 3
  • "When I inspect element" This is your error: you must view source instead to see what BeautifulSoup sees. It cannot run any client-side Javascript and cannot determine what is produced dynamically. – Karl Knechtel Jan 24 '22 at 00:48
  • I'm confused. If the class is `jobTitle jobTitle-newJob`, why did you tell the code to look for `jobTitle jobTitle-color-purple jobTitle-newJob`? – John Gordon Jan 24 '22 at 00:50
  • @JohnGordon I looked up "how to webscrape Indeed 2021" and found a youtube video which had this code. I wouldn't have known to include -color-purple just based on the case I was viewing. That's why I was confused on why I needed to include "-color-purple" for it to work. – AceCharm101 Jan 24 '22 at 01:05
  • @KarlKnechtel How do I view source? When I right-click Inspect I see the Elements tab opens up and I can also see the Sources tab. But when I click on Sources it doesn't allow me to hover over the element I want to pull. – AceCharm101 Jan 24 '22 at 01:13
  • It should be in the same right-click context menu as the Inspect option, or else in a menu option, depending on your browser. – Karl Knechtel Jan 24 '22 at 07:01

0 Answers0