1

I am looking to use a crawler to fetch data from a site, I found How do I make a simple crawler in PHP? and it was helpfull but I am looking to use the code on http://findpeopleonplus.com/ to get all the google plus links from the pages.

I will paste the code here for reference : 

$seen[$url] = true;

$dom = new DOMDocument('1.0');
@$dom->loadHTMLFile($url);

$anchors = $dom->getElementsByTagName('a');
foreach ($anchors as $element) {
    $href = $element->getAttribute('href');
    if (0 !== strpos($href, 'http')) {
        $path = '/' . ltrim($href, '/');
        if (extension_loaded('http')) {
            $href = http_build_url($url, array('path' => $path));
        } else {
            $parts = parse_url($url);
            $href = $parts['scheme'] . '://';
            if (isset($parts['user']) && isset($parts['pass'])) {
                $href .= $parts['user'] . ':' . $parts['pass'] . '@';
            }
            $href .= $parts['host'];
            if (isset($parts['port'])) {
                $href .= ':' . $parts['port'];
            }
            $href .= $path;
        }
    }
    crawl_page($href, $depth - 1);
}
echo "URL:",$url,PHP_EOL,"CONTENT:",PHP_EOL,$dom->saveHTML(),PHP_EOL,PHP_EOL;
}
crawl_page("http://hobodave.com", 2);
Community
  • 1
  • 1
ahoura
  • 689
  • 1
  • 6
  • 16
  • What exactly are you trying to accomplish and what is failing? – Rusty Fausak Sep 23 '11 at 23:22
  • well I am trying to crawl through the site, get EVERY SINGLE google plus link (it would be in this format : http://plus.google.com) and store it in a DB. nothing is failing, I am just not sure how to customize this function to get the links out so was hoping to get some help here – ahoura Sep 23 '11 at 23:25
  • Looks like you should do a regex or string match on the `$href` variable after `$href = $element->getAttribute('href');` – Rusty Fausak Sep 23 '11 at 23:29

0 Answers0