0

I am trying to scrape news articles using Beautiful Soup. However, it only works for some articles on the website and not for others. I can't find any apparent differences in the source code so I would be very grateful for any ideas on how to solve this.

For example, this is fine:

import requests
from bs4 import BeautifulSoup

result = requests.get("https://www.dn.se/nyheter/sverige/ewa-stenberg-darfor-ligger-sverige-steget-efter/")
src = result.content
soup = BeautifulSoup(src, 'lxml')

content = soup.find('div', class_='article__body')
body = content.text
print(body)

But changing the url to:

result = requests.get("https://www.dn.se/nyheter/sverige/Regeringen-vill-att-skolor-ska-fa-satta-betyg-i-arskurs-4")

produces the following error:

AttributeError: 'NoneType' object has no attribute 'text'

Tyra
  • 1

1 Answers1

1

In this example there is nothing wrong with the scraping itself, the second URL responds with a 301 (moved permanently redirect) which means you will get a new URL in the response. In requests you need to do some actions in order to follow the redirect.

See this answer https://stackoverflow.com/a/50606372/10201813 for info regarding how to solve it or read on http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history for more information

perlindstroem
  • 131
  • 6