How do I extract all URL links from an RSS feed?

Question

I need to extract all the links to news articles from the NY Times RSS feed to a MySQL database periodically. How do I go about doing this? Can I use some regular expression (in PHP) to match the links? Or is there some other alternative way? Thanks in advance.

Answer 1

UPDATE 2 I tested the code below and had to modify the

    $links = $dom->getElementsByTagName('a'); 

and change it to:

    $links = $dom->getElementsByTagName('link');

It successfully outputted the links. Good Luck

---

UPDATE Looks like there is a complete answer here: How do you parse and process HTML/XML in PHP.

I developed a solution so that I could recurse all the links in my website. I've removed the code which verified the domain was the same with each recursion (since the question didn't ask for this), but you can easily add one back in if you need it.

---

Using html5 DOMDocument, you can parse HTML or XML document to read links. It is better than using regex. Try something like this

<?php
//300 seconds = 5 minutes - or however long you need so php won't time out
ini_set('max_execution_time', 300); 

// using a global to store the links in case there is recursion, it makes it easy. 
// You could of course pass the array by reference for cleaner code.
$alinks = array();

// set the link to whatever you are reading
$link = "http://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml";

// do the search
linksearch($link, $alinks);

// show results
var_dump($alinks);

function linksearch($url, & $alinks) {
    // use $queue if you want this fn to be recursive
    $queue = array();
    echo "<br>Searching: $url";

    $href = array();
    //Load the HTML page
    $html = file_get_contents($url);

    //Create a new DOM document
    $dom = new DOMDocument;

    //Parse the HTML. The @ is used to suppress any parsing errors
    //that will be thrown if the $html string isn't valid XHTML.
    @$dom->loadHTML($html);

    //Get all links. You could also use any other tag name here,
    //like 'img' or 'table', to extract other tags.
    $links = $dom->getElementsByTagName('link');

    //Iterate over the extracted links and display their URLs
    foreach ($links as $link){

        //Extract and show the "href" attribute. 
        $href[] = $link->getAttribute('href');
    }    
    foreach (array_unique($href) as $link) {            
        // add to list of links found
        $queue[] = $link;
    }

    // remove duplicates
    $queue = array_unique($queue);

    // get links that haven't yet been processed
    $queue = array_diff($queue, $alinks);

    // update array passed by reference with new links found
    $alinks = array_merge($alinks, $queue);

    if (count($queue) > 0) {
        foreach ($queue as $link) {
            // recursive search - uncomment out if you use this
            // remember to check that the domain is the same as the one starting from
            // linksearch($link, $alinks);
        }
    }
}

Answer 2

DOM+Xpath allows you to fetch nodes using expressions.

RSS Item Links

To fetch the RSS link elements (the link for each item):

$xml = file_get_contents($url);    
$document = new DOMDocument();
$document->loadXml($xml);
$xpath = new DOMXPath($document);

$expression = '//channel/item/link';

foreach ($xpath->evaluate($expression) as $link) {
  var_dump($link->textContent);
}

Atom Links

The atom:link have a different semantic, they are part of the Atom namespace and used to describe relations. NYT uses the standout relation to mark featured stories. To fetch the Atom links you need to register a prefix for the namespace. Attributes are nodes, too so you can fetch them directly:

$xml = file_get_contents($url);
$document = new DOMDocument();
$document->loadXml($xml);
$xpath = new DOMXPath($document);
$xpath->registerNamespace('a', 'http://www.w3.org/2005/Atom');

$expression = '//channel/item/a:link[@rel="standout"]/@href';

foreach ($xpath->evaluate($expression) as $link) {
  var_dump($link->value);
}

Here are other relations like prev and next.

HTML Links (a elements)

The description elements contain HTML fragments. To extract the links from them you have to load the HTML into a separate DOM document.

$xml = file_get_contents($url);
$document = new DOMDocument();
$document->loadXml($xml);
$xpath = new DOMXPath($document);
$xpath->registerNamespace('a', 'http://www.w3.org/2005/Atom');

$expression = '//channel/item/description';

foreach ($xpath->evaluate($expression) as $description) {
  $fragment = new DOMDocument();
  $fragment->loadHtml($description->textContent);
  $fragmentXpath = new DOMXpath($fragment);
  foreach ($fragmentXpath->evaluate('//a[@href]/@href') as $link) {
    var_dump($link->value);
  } 
}