Newspaper library

Question

As an absolute newbie on the topic of using python, I stumbled over a few difficulties using the newspaper library extension. My goal is to use the newspaper extension on a regular basis to download all new articles of a German news website called "tagesschau" and all articles from CNN to build a data stack I can analyze in a few years. If I got it right I could use the following commands to download and scrape all articles into the python library.

import newspaper
from newspaper import news_pool

tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')

papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
news_pool.join()`

If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?

Thanks for your help.

Mrinal Roy · Accepted Answer · 2018-11-16T08:57:21.510

1

The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....

import newspaper
from newspaper import news_pool

tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')

papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2)
news_pool.join()

for i in range (tagesschau_paper.size()): 
    with open("tagesschau_paper{}.html".format(i), "w") as file:
    file.write(tagesschau_paper.articles[i].html)

Note: news_pool doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size(), it results to 0. You have to import and use Source instead.

The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.

edited Nov 16 '18 at 08:57

answered Nov 16 '18 at 08:49

Mrinal Roy

969
7
12

Thanks for youre help :) – Philipp Schulz Nov 16 '18 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:` – Philipp Schulz Nov 18 '18 at 11:53
Thanks for the update. I didn't need to set the encoding. – Mrinal Roy Nov 21 '18 at 02:20

score -1 · Answer 2 · answered Nov 13 '18 at 22:41

-1

You can use pickle to save objects outside of python and reopen them later:

file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb') 

# this writes the object news_pool to the
# file named 'testfile'
pickle.dump(news_pool,fileObject)   

# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')  
# load the object from the file into var news_pool_reopen
news_pool_reopen = pickle.load(fileObject)

answered Nov 13 '18 at 22:41

Turtalicious

430
2
5

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of `news_pool` changes (which seems likely given the time span OP specified and the fact that `newspaper` seems to be in active development), then `pickle.load` will likely fail. See, for example, [this](https://stackoverflow.com/questions/10842553/pickle-with-custom-classes) explanation. – Paul Brodersen Nov 13 '18 at 23:07

Newspaper library

2 Answers2