I want to extract the title of an article using BeautifulSoup 4.
Here's my scrape function:
def scrape(request):
session = requests.Session()
session.headers = {"User-Agent": "Googlebot/2.1 (+http://www.google.com/bot.html)"}
url = "https://www.ynet.co.il/home/0,7340,L-2,00.html"
content = session.get(url, verify=False).content
soup = BSoup(content, "html.parser")
News = soup.find_all('div', {"class":"str3s str3s_small str3s_type_small"})
for artcile in News:
main = artcile.find_all('a')[0]
link = main['href']
image_src = str(main.find('img')['src']).split(" ")[0]
title = main['title']
new_headline = Headline()
new_headline.title = title
new_headline.url = link
new_headline.image = image_src
new_headline.save()
return redirect("../")
As you can see my title is set to main['title']
, but the problem is that this ['title']
searches for title elements and what I'm looking for is, for example, a
<div class title>title</div>
.
How do I do that?