Processing HTML files Python

Question

I dont know much about html... How do you remove just text from the page? For example if the html page reads as:

<meta name="title" content="How can I make money at home online? No gimmacks please? - Yahoo! Answers">
<title>How can I make money at home online? No gimmicks please? - Yahoo! Answers</title>

I just want to extract this.

How can I make money at home online? No gimmicks please? - Yahoo! Answers

I am using re function:

def striphtml(data):
  p = re.compile(r'<.*?>')
  return p.sub(' ',data)

but still it's not doing what I intend it to do..?

The above function is called as:

for lines in filehandle.readlines():

        #k = str(section[6].strip())
        myFile.write(lines)

        lines = striphtml(lines)
        content.append(lines)

possible duplicate of [Parsing HTML in Python](http://stackoverflow.com/questions/717541/parsing-html-in-python), [Processing a HTML file using Python](http://stackoverflow.com/q/7694637) — Sathyajith Bhat, Jan 09 '12 at 02:45
Check this question: http://stackoverflow.com/questions/328356/extracting-text-from-html-file-using-python — mgibsonbr, Jan 09 '12 at 02:47

Fabián Heredia Montiel · Accepted Answer · 2012-01-09T02:54:25.340

2

Don't use Regular expressions for HTML/XML parsing. Try http://www.crummy.com/software/BeautifulSoup/ instead.

from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup('Your resource<title>hi</title>')
soup.title.string # Your title string.

edited Jan 09 '12 at 02:54

answered Jan 09 '12 at 02:47

Fabián Heredia Montiel

1,687
1
16
30

1

Update: try `from bs4 import BeautifulSoup` – volvox Dec 10 '22 at 09:38

score 2 · Answer 2 · answered Jan 09 '12 at 02:58

2

Use an html parser for that. One could be BeautifulSoup

To get text content of the page:

 from BeautifulSoup import BeautifulSoup


 soup = BeautifulSoup(your_html)
 text_nodes = soup.findAll(text = True)
 retult = ' '.join(text_nodes)

answered Jan 09 '12 at 02:58

soulcheck

36,297
6
91
90

score 1 · Answer 3 · answered Jan 09 '12 at 02:56

I usually use http://lxml.de/ for html parsing! it is really easy to use, and pretty much to get tags you can use xpath for it! which just make things easy as well as fast.

I have a example of use, in a script that I did to read a xml feed and count the words:

https://gist.github.com/1425228

Also you can find more examples in the documentation: http://lxml.de/lxmlhtml.html

Processing HTML files Python

3 Answers3

Linked