How to fix Newspaper3k 403 Client Error for certain URL's?

Question

I am trying to get a list of articles using a combo of the googlesearch and newspaper3k python packages. When using article.parse, I end up getting an error: newspaper.article.ArticleException: Article download() failed with 403 Client Error: Forbidden for url: https://www.newsweek.com/donald-trump-hillary-clinton-2020-rally-orlando-1444697 on URL https://www.newsweek.com/donald-trump-hillary-clinton-2020-rally-orlando-1444697

I have tried running as admin when executing script and the link works when opening straight in a browser.

Here is my code:

import googlesearch
from newspaper import Article

query = "trump"
urlList = []

for j in googlesearch.search_news(query, tld="com", num=500, stop=200, pause=.01):
    urlList.append(j)

print(urlList)

articleList = []

for i in urlList:
    article = Article(i)
    article.download()
    article.html
    article.parse()
    articleList.append(article.text)
    print(article.text)

Here is my full error output:

Traceback (most recent call last):
  File "C:/Users/andre/PycharmProjects/StockBot/WebCrawlerTest.py", line 31, in <module>
    article.parse()
  File "C:\Users\andre\AppData\Local\Programs\Python\Python37\lib\site-packages\newspaper\article.py", line 191, in parse
    self.throw_if_not_downloaded_verbose()
  File "C:\Users\andre\AppData\Local\Programs\Python\Python37\lib\site-packages\newspaper\article.py", line 532, in throw_if_not_downloaded_verbose
    (self.download_exception_msg, self.url))
newspaper.article.ArticleException: Article `download()` failed with 403 Client Error: Forbidden for url: https://www.newsweek.com/donald-trump-hillary-clinton-2020-rally-orlando-1444697 on URL https://www.newsweek.com/donald-trump-hillary-clinton-2020-rally-orlando-1444697

I expected it to just output the text of the article. Any help you can give would be great. Thanks!

You have probably been blacklisted by the server (newsweek.com). — Selcuk, Jun 20 '19 at 04:02
Even though it works through a regular browser? Is there a way around this that you know of? Thanks! — totalmayhem, Jun 21 '19 at 02:42

score 10 · Accepted Answer · answered Jun 26 '19 at 18:22

10

I got it to work by changing the user-agent

from newspaper import Article
from newspaper import Config

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
config = Config()
config.browser_user_agent = user_agent
page = Article("https://www.newsweek.com/donald-trump-hillary-clinton-2020-rally-orlando-1444697", config=config)
page.download()
page.parse()
print(page.text)

answered Jun 26 '19 at 18:22

Ashish Cherian

367
1
3
15

2

I still get an error newspaper.article.ArticleException: Article `download()` failed with 403 Client Error: Forbidden for url: https://www.newsweek.com/new-mexico-compound-charges-dropped-children-1096830 on URL https://www.newsweek.com/new-mexico-compound-charges-dropped-children-1096830 – Mona Jalal Jul 23 '20 at 17:53
They have installed a firewall - that returns a captcha. The above should still work for most websites. `nNeed permission to access data? Contact: DataAccess@datadome.co\n-->\` – Ashish Cherian Jul 24 '20 at 19:30
can you give more info @Cerin – Ashish Cherian Feb 22 '21 at 18:49
@AshishCherian Nevermind. Turns out my URL was http and the server wanted me to use https, however, it was returning a 403 error instead of the proper 301 redirect. – Cerin Feb 23 '21 at 01:30

How to fix Newspaper3k 403 Client Error for certain URL's?

1 Answers1

Linked