What are good Perl or Python starting points for a site scraping library?

Question

Possible Duplicate:
How can I screen scrape with Perl?
Web scraping with Python

This isn't my field of work, so pardon the general lack of knowledge. I'm looking for a Python or Perl library for site scraping (getting some products information from a site / tables on various pages / into a more user friendly format - Excel - for which both languages have satisfactory options) with good documentation.

Can anybody give a recommendation or a starting point on the subject? Googling gives several interesting matches, but having a short bit of time I'd rather not go hunting on a wrong track, but would rather trust someone with some experience in the matter.

Answer 1

In python there is a library called scrapy as well as more basic ones such as using mechanize or other interface with a parser such as lxml or beautifulsoup

In the comments it was mentioned that they do not have tutorials, but using mechanize is relatively simple (using its browser object) while lxml provides an easy way to jump around the dom using xpath.

Although I have never used it, Selenium also seems like a good option, albeit much more complicated

Answer 2

I needed to hunt down all instances of a pesky HTML class a few days ago, and threw together the following in next to no time - it's both a scraper and a crawler, and it's tiny.

import sys
import urllib.parse as uparse
import httplib2
from bs4 import BeautifulSoup

http = httplib2.Http()

hit_urls = set()

def crawl(url, check, do, depth=1):
    global hit_urls
    if url in hit_urls:
        #print("**Skipping %s" % url)
        return
    #print("Crawling %s" % url, file=sys.stderr)
    hit_urls.add(url)

    _, response = http.request(url)
    soup = BeautifulSoup(response)

    resp = do(url, soup)

    if depth > 0:
        for link in soup.find_all('a'):
            if link.has_key('href'):
                rel_url = link['href']
                if(check(rel_url)):
                    crawl(uparse.urljoin(url,rel_url), check, do, depth-1)

    return resp

def isLocal(url):
    if not url.startswith('/'):
        return False
    if url.startswith('/goToUrl'): # 3rd party redirect page
        return False
    return True

def findBadClass(url, soup):
    for t in soup.find_all(True,{'class': 'badClass'}):
        print(url+":"+str(t))

if __name__ == '__main__':
    crawl('http://example.com', isLocal, findBadClass)

Answer 3

If you just want to scrape a handful of sites with consistent formatting, the easiest thing would probably be to use requests combined with regular expressions and python's built-in string processing.

import re

import requests


resp = requests.get('http://austin.craigslist.org/cto/')

regex = ('<a href="(http://austin.craigslist.org/cto/[0-9]+\.html)">'
         '([a-zA-z0-9 ]+)</a>')

for i, match in enumerate(re.finditer(regex, resp.content)):
    if i > 5:
        break
    url = match.group(1)
    print 'url:', url
    resp = requests.get(url)
    title = re.search('<h2>(.+)</h2>', resp.content).group(1)
    print 'title:', title
    body = resp.content.split('<div id="userbody">', 1)[1]
    body = body.split('<script type="text/javascript">')[0]
    body = body.split('<!-- START CLTAGS -->')[0]
    print 'body:', body
    print

Edit: To clarify, I've used Beautiful Soup and think it's overrated. I thought it was weird and wonky and hard to use in real-world circumstances. Also, it's too much work to learn a new library for a one-off scraper -- you're better off using standard techniques (ie the ones I suggested above) that can be applied elsewhere when doing python scripting.