PHP web crawler, data structure and storage, Will it work with PHPCrawl?

Question

If there are other classes written to do this, a link would be awesome. If not, how can I do it with PHPCrawl?

Is it possible to store specific information from a crawled site based upon a set of rules specific to the site? Ex., [div.wantThis, img#defaultPicture] is the array returned for site A and only [div.shortTextContent] is the array returned for site B?

In PHPCrawl, how can I get this information out of the $page_data array?

Needs

Must be able to target only certain elements.

Able to read the data storage rule from a variable (which could be an array specifying the element(s) to target).

score 0 · Accepted Answer · answered Nov 29 '11 at 09:30

What you are asking is how to parse specific content from site A and some other specific content from site B using PHPCrawl.

For site specific parsing style following if-else approach can be followed:

for url in urls:
    content = crawl(url)
    if(url of type 1?):
        extract_style1(content)
    else-if(url of type 2?):
        extract_style2(content)
    else:
        extract_styledefault(content)

For specific content extracting following algo can be used:

Note: There are spectrum of parsing techniques avaliable, I am implmeneting HTML DOM Parsing here..

// Create DOM from your PHP Crawl Data Source
$html = $page_data[source]

// Find all images 
foreach($html->find('img') as $element) 
       echo $element->src . '<br>';

// Find all links 
foreach($html->find('a') as $element) 
       echo $element->href . '<br>';

Reference:

HTML DOM
PHPCrawl Example

PHP web crawler, data structure and storage, Will it work with PHPCrawl?

1 Answers1