How to get all urls from page (php)

Question

I have a page with urls with descriptions listed one under another (something like bookmarks/list of sites). How do I use php to get all urls from that page and write them to txt file (one per line, only url without description)?

Page looks like this:

Some description

Other description

Another one

And I would like script's txt output to look like this:

http://link.com

http://link2.com

http://link3.com

score 16 · Accepted Answer · answered Jul 15 '09 at 00:34

16

one way

$url="http://wwww.somewhere.com";
$data=file_get_contents($url);
$data = strip_tags($data,"<a>");
$d = preg_split("/<\/a>/",$data);
foreach ( $d as $k=>$u ){
    if( strpos($u, "<a href=") !== FALSE ){
        $u = preg_replace("/.*<a\s+href=\"/sm","",$u);
        $u = preg_replace("/\".*/","",$u);
        print $u."\n";
    }
}

answered Jul 15 '09 at 00:34

ghostdog74

327,991
56
259
343

2

if my link like this : Above code not find link – Bharat Dangar Sep 22 '17 at 05:45
2

That's because user2066719's answer is the correct one (at least in 2018). You shouldn't be using regular expressions to search HTML/XML documents- they're too brittle, and there are much more powerful tools even in native PHP these days. – Evan Mar 13 '18 at 01:43

score 14 · Answer 2 · answered Mar 14 '13 at 16:17

Another way

$url = "http://wwww.somewhere.com";

$html = file_get_contents($url);

$doc = new DOMDocument();
$doc->loadHTML($html); //helps if html is well formed and has proper use of html entities!

$xpath = new DOMXpath($doc);

$nodes = $xpath->query('//a');

foreach($nodes as $node) {
    var_dump($node->getAttribute('href'));
}

score 6 · Answer 3 · answered Apr 30 '16 at 13:47

You can use this to get all the link in the given web page.

<?php

    $var = fread_url($url);

    preg_match_all ("/a[\s]+[^>]*?href[\s]?=[\s\"\']+".
                    "(.*?)[\"\']+.*?>"."([^<]+|.*?)?<\/a>/", 
                    $var, &$matches);

    $matches = $matches[1];
    $list = array();

    foreach($matches as $var)
    {    
        print($var."<br>");
    }

    function fread_url($url,$ref="")
    {
        if(function_exists("curl_init")){
            $ch = curl_init();
            $user_agent = "Mozilla/4.0 (compatible; MSIE 5.01; ".
                          "Windows NT 5.0)";
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
            curl_setopt( $ch, CURLOPT_HTTPGET, 1 );
            curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
            curl_setopt( $ch, CURLOPT_FOLLOWLOCATION , 1 );
            curl_setopt( $ch, CURLOPT_FOLLOWLOCATION , 1 );
            curl_setopt( $ch, CURLOPT_URL, $url );
            curl_setopt( $ch, CURLOPT_REFERER, $ref );
            curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
            $html = curl_exec($ch);
            curl_close($ch);
        }
       else{
            $hfile = fopen($url,"r");
            if($hfile){
                while(!feof($hfile)){
                    $html.=fgets($hfile,1024);
                }
            }
        }
        return $html;
    }

    ?>

score 0 · Answer 4 · answered Feb 10 '23 at 04:32

To obtain all the URLs of the page, obtain all the links and download the file.

<?php
$host = "http://urlname/";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $host);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, "http://192.168.2.104/filetest/");
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
curl_close($ch);   


$files = array();
$data  = strip_tags($result,"<a>");
$d     = preg_split("/<\/a>/",$data);

foreach ( $d as $k=>$u ){
 if( strpos($u, "<a href=") !== FALSE )
 {
    $u = preg_replace("/.*<a\s+href=\"/sm","",$u);
    $u = preg_replace("/\".*/","",$u);
    if (strpos($u , ".") !== false) 
    {
        array_push($files,urldecode($u));
    }        
 }
}


foreach ($files as $filenm) {
  $url = "http://urlname/".$filenm;
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');

  $result = curl_exec($ch);
  if (curl_errno($ch)) {
     echo 'Error:' . curl_error($ch);
  }
  curl_close($ch);

  file_put_contents("downloadfile/".$filenm, $result);
 }
 ?>

How to get all urls from page (php)

4 Answers4

Linked