Parse Website for URLs

Question

Just wondering if someone can help me further with the following. I want to parse the URL on this website:http://www.directorycritic.com/free-directory-list.html?pg=1&sort=pr

I have the following code:

<?PHP  
$url = "http://www.directorycritic.com/free-directory-list.html?pg=1&sort=pr";
$input = @file_get_contents($url) or die("Could not access file: $url"); 
$regexp = "<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>"; 
if(preg_match_all("/$regexp/siU", $input, $matches)) { 
// $matches[2] = array of link addresses 
// $matches[3] = array of link text - including HTML code
} 
?>

Which does nothing at present and what I need this to do is scrap all the URL in the table for all 16 pages and would really appreciate some help with how to amend the above to do that and output URL into a text file.

*(related)* [Best Methods to parse HTML](http://stackoverflow.com/questions/3577641/best-methods-to-parse-html/3577662#3577662) — Gordon, Dec 16 '10 at 13:29
possible duplicate of [Regular expression for grabbing the href attribute of an A element](http://stackoverflow.com/questions/3820666/regular-expression-for-grabbing-the-href-attribute-of-an-a-element) — Gordon, Dec 16 '10 at 13:31

score 5 · Answer 1 · edited May 23 '17 at 11:47

5

Use HTML Dom Parser

$html = file_get_html('http://www.example.com/');

// Find all links
$links = array(); 
foreach($html->find('a') as $element) 
       $links[] = $element->href;

Now links array contains all URLs of given page and you can use these URLs to parse further.

Parsing HTML with regular expressions is not a good idea. Here are some related posts:

EDIT:

Some Other HTML Parsing tools as described by Gordon in comments below:

edited May 23 '17 at 11:47

Community

1
1

answered Dec 16 '10 at 13:22

Naveed

41,517
32
98
131

1

Suggested third party alternatives to [SimpleHtmlDom](http://simplehtmldom.sourceforge.net/) that actually use [DOM](http://php.net/manual/en/book.dom.php) instead of String Parsing: [phpQuery](http://code.google.com/p/phpquery/), [Zend_Dom](http://framework.zend.com/manual/en/zend.dom.html), [QueryPath](http://querypath.org/) and [FluentDom](http://www.fluentdom.org). – Gordon Dec 16 '10 at 13:28
@Gordon: Yes you are right. I think we both have answered same type of question before. – Naveed Dec 16 '10 at 13:33

score 3 · Answer 2 · edited Oct 07 '21 at 05:59

You really shouldn’t use regular expressions to parse HTML as it’s to error prone.

Better use an HTML parser like the one of PHP’s DOM library:

$code = file_get_contents($url);
$doc = new DOMDocument();
$doc->loadHTML($code);
$links = array();
foreach ($doc->getElementsByTagName('a') as $element) {
    if ($element->hasAttribute('href')) {
        $links[] = $elements->getAttribute('href');
    }
}

Note that this will collect the URI references as they appear in the document and not as an absolute URI. You might want to resolve them before.

It seems that PHP doesn’t provide an appropriate library (or I haven’t found it yet). But see RFC 3986 – Reference Resolution and my answer on Convert a relative URL to an absolute URL with Simple HTML DOM? for further details.

score 0 · Answer 3 · answered Jan 14 '20 at 12:51

Try this method

function getinboundLinks($domain_name) {
ini_set('user_agent', 'NameOfAgent (<a class="linkclass" href="http://localhost">http://localhost</a>)');
 $url = $domain_name;
$url_without_www=str_replace('http://','',$url);
$url_without_www=str_replace('www.','',$url_without_www);
 $url_without_www= str_replace(strstr($url_without_www,'/'),'',$url_without_www);
$url_without_www=trim($url_without_www);
$input = @file_get_contents($url) or die('Could not access file: $url');
 $regexp = "<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";
//$inbound=0;
$outbound=0;
$nonfollow=0;
if(preg_match_all("/$regexp/siU", $input, $matches, PREG_SET_ORDER)) {
foreach($matches as $match) {
# $match[2] = link address
 # $match[3] = link text
//echo $match[3].'<br>';
if(!empty($match[2]) && !empty($match[3])) {
if(strstr(strtolower($match[2]),'URL:') || strstr(strtolower($match[2]),'url:') ) {
$nonfollow +=1;
} else if (strstr(strtolower($match[2]),$url_without_www) || !strstr(strtolower($match[2]),'http://')) {
     $inbound += 1;
    echo '<br>inbound '. $match[2];
 }
else if (!strstr(strtolower($match[2]),$url_without_www) && strstr(strtolower($match[2]),'http://')) {
echo '<br>outbound '. $match[2];
     $outbound += 1;
    }
}
}
}
$links['inbound']=$inbound;
$links['outbound']=$outbound;
$links['nonfollow']=$nonfollow;
return $links;
}

// ************************Usage********************************
$Domain='<a class="linkclass" href="http://zachbrowne.com">http://zachbrowne.com</a>';
$links=getinboundLinks($Domain);
echo '<br>Number of inbound Links '.$links['inbound'];
echo '<br>Number of outbound Links '.$links['outbound'];
echo '<br>Number of Nonfollow Links '.$links['nonfollow'];

Parse Website for URLs

3 Answers3

Linked

Related