39

I've been doing some HTML scraping in PHP using regular expressions. This works, but the result is finicky and fragile. Has anyone used any packages that provide a more robust solution? A config driven solution would be ideal, but I'm not picky.

Ijas Ameenudeen
  • 9,069
  • 3
  • 41
  • 54
tsellon
  • 2,396
  • 5
  • 24
  • 33
  • Have a look at [this](http://stackoverflow.com/questions/26947/how-to-implement-a-web-scraper-in-php#27109) thread - the question goes into a similar direction – crono Aug 29 '08 at 08:16

7 Answers7

28

I would recomend PHP Simple HTML DOM Parser after you have scraped the HTML from the page. It supports invalid HTML, and provides a very easy way to handle HTML elements.

Espo
  • 41,399
  • 21
  • 132
  • 159
  • 8
    Suggested third party alternatives to [SimpleHtmlDom](http://simplehtmldom.sourceforge.net/) that actually use [DOM](http://php.net/manual/en/book.dom.php) instead of String Parsing: [phpQuery](http://code.google.com/p/phpquery/), [Zend_Dom](http://framework.zend.com/manual/en/zend.dom.html), [QueryPath](http://querypath.org/) and [FluentDom](http://www.fluentdom.org). – Gordon Oct 10 '11 at 15:08
  • can you give me example to click on any link on a given page? – sagar junnarkar Nov 12 '13 at 07:51
5

I would also recommend 'Simple HTML DOM Parser.' It is a good option particularly if your familiar with jQuery or JavaScript selectors then you will find yourself at home.

I have even blogged about it in the past.

5

If the page you're scraping is valid X(HT)ML, then any of PHP's built-in XML parsers will do.

I haven't had much success with PHP libraries for scraping. If you're adventurous though, you can try simplehtmldom. I'd recommend Hpricot for Ruby or Beautiful Soup for Python, which are both excellent parsers for HTML.

John Douthat
  • 40,711
  • 10
  • 69
  • 66
  • If you're going to be parsing particularly sloppy HTML, make sure you don't use BeautifulSoup 3.1.x (use 3.0.x). 3.1.x uses htmllib as its parser, which is much less forgiving than 3.0.x's use of sgmllib. – Tom Mar 18 '09 at 01:33
5

I had some fun working with htmlSQL, which is not so much a high end solution, but really simple to work with.

BlaM
  • 28,465
  • 32
  • 91
  • 105
3

Using PHP for HTML scraping, I'd recommend cURL + regexp or cURL + some DOM parsers though I personally use cURL + regexp. If you have a profound taste of regexp, it's actually more accurate sometimes.

datasn.io
  • 12,564
  • 28
  • 113
  • 154
2

I've had very good with results with the Simple Html DOM Parser mentioned above as well. And then there's the  tidy Extension for PHP as well which works really well too.

Jan Gorman
  • 1,004
  • 2
  • 11
  • 16
2

I had to use curl on my host 1and1.

http://www.quickscrape.com/ is what I came up with using the Simple DOM class!

Steve
  • 21
  • 1