Trouble Getting a clean text file from HTML

Question

I have looked at these previous questions

I am trying to consolidate news and notes from websites.

Reputed News service websites allow Users to post comments and views.

I am trying to get only the news content without the users comments. I tried working with BeautifulSoup and html2text. But user-comments are being included in the text file. I have even tried developing a custom program but with no useful progress than the above two.

Can anybody provide some clue how to proceed?

The code:

import urllib2
from bs4 import BeautifulSoup
URL ='http://www.example.com'
print 'Following: ',URL
print "Loading..."
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
identify_as = { 'User-Agent' : user_agent }
print "Reading URL:"+str(URL)    
def process(URL,identify_as):
    req = urllib2.Request(URL,data=None,headers=identify_as)
    response = urllib2.urlopen(req) 
    _BSobj = BeautifulSoup(response).prettify(encoding='utf-8')
    return _BSobj #return beauifulsoup object
print 'Processing URL...'
new_string = process(URL,identify_as).split()

print 'Buidiing requested Text'
tagB = ['<title>','<p>']    
tagC = ['</title>','</p>']
reqText = []
for num in xrange(len(new_string)):
    buffText = [] #initialize and reset
    if new_string[num] in tagB: 
        tag = tagB.index(new_string[num])
        while new_string[num] != tagC[tag]:
            buffText.append(new_string[num])
            num+=1
        reqText.extend(buffText)


reqText= ''.join(reqText)
fileID = open('reqText.txt','w')
fileID.write(reqText)
fileID.close()

You'd better give us some samples. – zhangyangyu Jul 10 '13 at 15:25 — zhangyangyu, Jul 10 '13 at 15:25

score 1 · Accepted Answer · answered Jul 10 '13 at 15:36

Here's a quick example I wrote using urllib which gets the contents of a page to a file:

import urllib
import urllib.request
myurl = "http://www.mysite.com"

sock = urllib.request.urlopen(myurl)
pagedata = str(sock.read())                          
sock.close()

file = open("output.txt","w")
file.write(pagedata)
file.close()

Then with a lot of string formatting you should be able to extract the parts of the html you want. This gives you something to get started from.

Trouble Getting a clean text file from HTML

1 Answers1