scrape contents from site pages

Question

i got some problem and need help ..

my plan : 1. get ip from free proxy ( hi d e my ass ) 2. convert to XML

$html = file_get_contents('http://www.hidemyass.com/proxy-list/');

//$body = explode('<tbody>', $html);
$body = $html;


$xml = simplexml_load_string("<?xml version='1.0' encoding='utf-8'?><xml />");

$rows = array();
foreach (array_slice(explode('<td>', end($body)), 1) as $row)
{
    preg_match('/span>([0-9])<\/span>/', $row, $ids);
    preg_match('/span>([0-9])<\/span>/', $row, $dir);
    preg_match('/span>([0-9])<\/span>/', $row, $due);


    $node = $xml->addChild('train');

    $node->addChild('route', $ids[1]);
    $node->addChild('direction', $dir[1]);
    $node->addChild('due', $due[1]);
}

header('Content-Type: text/xml');
echo $xml->asXML();

but still can not..

can you help me ?

thanks jk

Don't use regexes for parsing html. http://stackoverflow.com/a/1732454/118068 Use DOM instead. — Marc B, Apr 09 '12 at 18:35

score 6 · Accepted Answer · edited Jan 24 '14 at 11:49

The simplest and ideal solution would have been simple_html_dom please see : http://simplehtmldom.sourceforge.net/

Example

    include 'simple_html_dom.php';
    $html = file_get_html('http://www.hidemyass.com/proxy-list/');
    echo "<pre>";
    foreach ( $html->find ( 'tr' ) as $element ) {
        $ip = $element->find ( 'td', 1 );
        $port = $element->find ( 'td', 2 );
        $ip = getIP ( $ip );
        // var_dump($element->xmltext);
        echo " $ip : $port \n";
    }

    function getIP($obj) {
        global $html;

        $text = str_replace ( "div", "span", $obj->xmltext );
        $text = explode ( "span", $text );

        $ip = array ();

        foreach ( $text as $value ) {
            $value = trim ( $value );
            $value = trim ( $value, "<" );
            $value = trim ( $value, ">" );
            $value = trim ( $value, "." );

            if (empty ( $value ))
                continue;

            if (strpos ( $value, "display:none" )) {
                continue;
            }

            if (strpos ( $value, ">" )) {
                $value = "<" . $value . ">";
            }

            $value = strip_tags ( $value );

            $value = trim ( $value, "." );

            if (empty ( $value ))
                continue;

            $ip [] = $value;
        }

        if (is_array ( $ip )) {
            return implode ( ".", $ip );
        }
    }

But this would not give you the IP address in the format you want because HideMyASS is protecting such kind of extraction

A td containing IP address would look like this

<td>
<span><span class="52">201</span>.73
    <div style="display: none">228</div>
    <span class="" style="">.</span><span>17</span><span
    style="display: none">248</span><span></span>.107</span>
</td>

Can you see <div style="display: none"> and sometimes <span style="display: none"> some tiles they use integer clases such as class=51 which also means none ....

I was able to get a crazy work aroud .. using getIP function .... i hope this helps

Output Example

 IP address : Port 
 200.135.197.120 :  8080 
 96.46.7.194 :  80 
 217.26.14.18 :  3128 
 189.114.111.190 :  8080 
 202.51.107.37 :  8080 
 128.208.04.198 :  2124 
 221.133.238.138 :  8080 
 41.215.247.146 :  8080 
 140.113.216.134 :  3128 
 190.211.132.33 :  8080 
 117.34.92.43 :  3128 
 118.97.235.234 :  3128 
 85.248.141.245 :  3128 
 203.223.47.119 :  3128 
 200.48.213.82 :  8080 
 217.112.128.247 :  80 
 114.134.76.27 :  8080 
 78.45.134.10 :  3128 
 77.78.197.15 :  8080 
 189.44.226.66 :  3128 
 124.195.124.166 :  8080 
 190.39.128.219 :  8080 
 222.42.45.51 :  3128 
 195.138.76.136 :  3128 
 115.249.252.235 :  8080 
 222.124.152.18 :  8080 
 190.255.39.147 :  3128 
 189.22.138.162 :  8080 
 217.146.208.162 :  8080 
 203.143.18.1 :  8080 
 210.57.215.130 :  80 
 190.98.166.106 :  3128 
 200.5.226.74 :  80 
 187.6.254.19 :  3128 
 177.36.242.57 :  8080 
 41.133.101.242 :  8080 
 201.87.208.66 :  8080 
 41.67.20.91 :  8080 
 118.192.1.168 :  3128 
 41.75.201.146 :  3128 
 61.166.144.69 :  8080 
 200.238.98.234 :  3128 
 110.52.11.220 :  80 
 125.67.230.192 :  8080 
 94.228.35.219 :  80 
 64.85.181.45 :  8080 
 222.169.15.234 :  8080 
 113.106.194.220 :  80 
 119.82.239.50 :  8080 
 117.27.139.17 :  80

Thanks

Working on a workaround .. i'll update again as soon as am done — Baba, Apr 09 '12 at 19:14
WOW>>>>>>>>>SUPER ...thanks alot...how im owe you Baba...how do I reply goodness to you on the forum here. I am a newbie in this forum.BTW thanks alot — kimpuler, Apr 10 '12 at 02:08
You are welcome anytime ....:D ... surprised no one responded to your request ... — Baba, Apr 10 '12 at 02:10
@Baba unfortunately, it doesn't work any more. will be great if you can update it, you will be my hero! thank you. — user2203703, Dec 01 '13 at 21:34
@user2203703 Its much more simpler just use `simplexml_load_string` in the `getIP` function parse `/.([a-z0-9_]+)\{([a-z\:]+)\}/i` with the style. I really don't want to update the code because i think it was changed because of this answer :) — Baba, Dec 02 '13 at 19:16

scrape contents from site pages

1 Answers1