1

After reading this forum I am not sure which method is best to extract sections of data into a CSV file I.e. Python/Beautiful Soup/html2text. Because of the large number of files, I want to try and write a script I can run within the Terminal.

Output: One CSV file, with lines of text and five columns of data. e.g. first and last line

100 2010-12-20 145 ABC 04110000

1 2010-11-10 133 DDD 041123847

Thanks!

Lulu
  • 13
  • 5

3 Answers3

4

I would recommend using BeautifulSoup. Something like this will do (completely untested). Read the documentation for more.

csvfile = open('dump.csv', 'w')
for file in glob.glob('*.html'):
    print 'Processing', file
    soup = BeautifulSoup(open(file).read())
    for tr in soup.findAll('tr'):
        print >>csvfile, ' '.join(tr.findAll('td'))
moinudin
  • 134,091
  • 45
  • 190
  • 216
  • Thanks Marcog! I read through the BeautifulSoup documentation and can now extract the text I need into a csv file. I am still trying to work out how to pull out the data into columns so that each file I open and run puts a new row of data into the csv file. – Lulu Dec 27 '10 at 00:00
  • @Lulu If I understand what you want, adding a `,` to the end of the `print` line and adding a `print >>csvfile` at the end of the outer loop should do the trick. – moinudin Dec 27 '10 at 00:21
1

I don't know if Python natively supports XPath, but if it does, you should do some research on that subject.

Another alternative solution would be regular expressions.

Mathias Lykkegaard Lorenzen
  • 15,031
  • 23
  • 100
  • 187
  • 4
    "Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp." http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 – jfs Dec 25 '10 at 05:44
0

I've modified my code to:

#!/usr/bin/env python
import glob
import codecs
from BeautifulSoup import BeautifulSoup
with codecs.open('dump2.csv', "w", encoding="utf-8") as csvfile:
        for file in glob.glob('*html*'):
            print 'Processing', file
            soup = BeautifulSoup(open(file).read())
            rows = soup.findAll('tr')
            for tr in rows:
                    cols = tr.findAll('td')
                    #print >> csvfile,"#".join(col.string for col in cols)
                    #print >> csvfile,"#".join(td.find(text=True))
                    for col in cols:
                            print >> csvfile, col.string
                    print >> csvfile, "==="
            print >> csvfile, "***"

The code now pulls out data with * and === separators I then use perl to put into a clean csv file. For some reason it does not pull out all the required data but "misses" some data, e.g the Address1 and Address 2 data + the Date&Time and Number at the start of the table do not come out.

Lulu
  • 13
  • 5