How do I extract the text content of an article site with Python 3?

Question

I've tried the following:

import urllib

link = 'https://automatetheboringstuff.com/chapter7/'
f = urllib.request.urlopen(link)
myfile = f.read()
print(myfile)

But that just seems to return the page's source rather than the text content.

You need `BeautifulSoup` for that – t.m.adam Apr 19 '17 at 13:20 — t.m.adam, Apr 19 '17 at 13:20
Is that correct `urllib.request.urlopen(link)` ? – bhansa Apr 19 '17 at 13:25 — bhansa, Apr 19 '17 at 13:25

score 1 · Accepted Answer · answered Apr 19 '17 at 13:32

If you only want to get chapter text, beautiful soup is your choice, I think.

In your case:

import requests
from bs4 import BeautifulSoup

res = requests.get('https://automatetheboringstuff.com/chapter7/')
soup = BeautifulSoup(res.text, 'html.parser')
print(soup.find('div', { "class" : "book" }).text)

How do I extract the text content of an article site with Python 3?

1 Answers1