How to read a web page in PHP?

Question

I'm trying to save some web pages to text files using PHP scripts.

How can I load a web page into a file buffer with PHP and remove HTML tags?

Answer 1

One way:

$url = "http://www.brothersoft.com/publisher/xtracomponents.html";
$page = file_get_contents($url);
$outfile = "xtracomponents.html";
file_put_contents($outfile, $page);

The code above is just an example and lacks any(!) error checking and handling.

Answer 2

As the other answers have said, either standard PHP stream functions or cURL is your best bet for retrieving the HTML. As for removing the tags, here are a couple approaches:

Option #1: Use the Tidy extension, if available on your server, to walk through the document tree recursively and return the text from the nodes. Something like this:

function textFromHtml(TidyNode $node) {
    if ($node->isText()) {
        return $node->value;
    } else if ($node->hasChildren()) {
        $childText = '';
        foreach ($node->child as $child)
           $childText .= textFromHtml($child);
        return $childText;
    }
    return '';
}

You might want something more sophisticated than that, e.g., that replaces <br /> tags (where $node->name == 'br') with newlines, but this will do for a start.

Then, load the text of the HTML into a Tidy object and call your function on the body node. If you have the contents in a string, use:

$tidy = new tidy();
$tidy->parseString($contents);
$text = textFromHtml($tidy->body());

Option #2: Use regexes to strip everything between < and >. You could (and probably should) develop a more sophisticated regex that, for example, matched only valid HTML start or end tags. Any errors in the synax of the page, like a stray angle bracket in body text, could mean garbage output if you aren't careful. This is why Tidy is so nice (it is specifically designed to clean up bad pages), but it might not be available.

Answer 3

I strongly recommend you to take a look at SimpleHTML DOM class;

SimpleHTML DOM Parser at SourceForge

With it you can search the DOM tree using css selectors like with jQuery's $() function or prototypeJS $$() function.

Although it works with file_get_contents() to get content of a web page, you can pass it HTML only with some cURL class of yours ( if you need to login etc. )