Running a simple php web crawler via Snoopy - http://turma.sourceforge.net/web/urlator/snoopy.html - but the Fetchlinks() function is having a bit of an issue. Normally fetchlinks() grabs all the links on a page and slots them nicely into an array. This is working fine for all http:// links, but any https:// links are tacked on to http:// links, and aren't put in their own slot in the array.
Example of broken https links:
[472] => http://www.vapetropolis.ca/returns/https://www.vapetropolis.ca/product-warranties [473] => http://www.vapetropolis.ca/vaporizer-parts/davinci-vaporizer/https://www.vapetropolis.ca/product-warranties/https://www.vapetropolis.ca/customer/account/
As you can see, they're tacked on to properly formed http:// links.
This is the code that produces this
function crawl_link($link, array &$sitewide_link_list) {
$snoopy = new Snoopy;
//Get all links from first page
$snoopy->fetchlinks($link);
$currPage = $snoopy->results;
//Add all links to global array
//First, filter the links
//Ensure all links are from correct domain
$parsedDomain = 'www.vapetropolis.ca';
if (is_array($currPage)) {
foreach ($currPage as $link) {
if ($parsedDomain == parse_url( $link, PHP_URL_HOST )) {
if (!in_array($link, $sitewide_link_list)) {
array_push($sitewide_link_list, $link);
}
}
}
}
}
//Start crawl
//Global array of links
$sitewide_link_list = array();
$checkList = array();
$done = false;
$domain = 'http://www.vapetropolis.ca';
crawl_link($domain, $sitewide_link_list);
print_r($sitewide_link_list);
Any clue what might be going on?
