Load All TXT Files In A External Directory

Question

So I need to load all the txt files in: http://orcahub.com/unchecked-proxy-list/ as one txt file and go into my server which is a different one to Orcahub;

For some reason it wont work. I cant get it to actually get the HTML to even do regex.

What I tried:

<?php

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, 'http://orcahub.com/unchecked-proxy-list'); 
curl_setopt($ch, CURLOPT_HEADER, FALSE); 
curl_setopt($ch, CURLOPT_NOBODY, FALSE); // remove body 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); 
$st = curl_exec($ch); 
//curl_close($ch); 

//preg_match_all("/(.*\.txt)/", $st, $out);

var_dump($ch);
?>

UPDATE: New issue, I get a Server Error 500 when I use the following script: UPDATE: Found out this issue was from a newline after the URL.

<?php

function disguise_curl($url) {

    //Prepare Curl;
    $curl = curl_init();

    //Setup Headers (Firefox 2.0.0.6);
    $header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,"; 
    $header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"; 
    $header[] = "Cache-Control: max-age=0"; 
    $header[] = "Connection: keep-alive"; 
    $header[] = "Keep-Alive: 300"; 
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7"; 
    $header[] = "Accept-Language: en-us,en;q=0.5"; 
    $header[] = "Pragma: ";

    //Setup Curl;
    curl_setopt($curl, CURLOPT_URL, $url); 
    curl_setopt($curl, CURLOPT_USERAGENT, 'Googlebot/2.1 (+http://www.google.com/bot.html)'); 
    curl_setopt($curl, CURLOPT_HTTPHEADER, $header); 
    curl_setopt($curl, CURLOPT_REFERER, 'http://orcahub.com/unchecked-proxy-list/'); 
    curl_setopt($curl, CURLOPT_ENCODING, 'gzip,deflate'); 
    curl_setopt($curl, CURLOPT_AUTOREFERER, true); 
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); 
    curl_setopt($curl, CURLOPT_TIMEOUT, 60); 

    //Execute Curl;
    $html = curl_exec($curl);

    //End Curl;
    curl_close($curl);

    //Output the HTML;
    return $html;

}

function rem_href($x) { return substr(strstr($x, '>'), strlen('>')); }

$response = disguise_curl('http://orcahub.com/unchecked-proxy-list/'); 
preg_match_all("/<a[\s]+[^>]*?href[\s]?=[\s\"\']+"."(.*?)[\"\']+.*?>"."([^<]+|.*?)?<\/a>/", $response, $matches, PREG_SET_ORDER );

foreach($matches as $value) { 
    $proxylists[] = 'http://orcahub.com/unchecked-proxy-list/'.rem_href($value[0]);
};

echo $proxylists[0];

$response = disguise_curl($proxylists[0]);
//Server Error 500 Here;
echo $response;

?>

Correct, I believe the server is blocking my requests somehow — John Butler, Dec 04 '15 at 18:04
You can use xpath to get all file names: http://stackoverflow.com/questions/10108634/crawl-a-website-get-the-links-crawl-the-links-with-php-and-xpath Then foreach txt use file() function - to get all lines as an array and with fopen append to your file — ka_lin, Dec 04 '15 at 18:05
http://toolnet.work/sslpro.xyz/getlists.php Doesnt seem to work — John Butler, Dec 04 '15 at 18:08

score 1 · Accepted Answer · answered Dec 04 '15 at 18:17

Came accross from php.net a function that add headers to disguise the call, a regex I added for parsing the response:

function disguise_curl($url) 
{ 
  $curl = curl_init(); 

  // Setup headers - I used the same headers from Firefox version 2.0.0.6 
  // below was split up because php.net said the line was too long. :/ 
  $header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,"; 
  $header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"; 
  $header[] = "Cache-Control: max-age=0"; 
  $header[] = "Connection: keep-alive"; 
  $header[] = "Keep-Alive: 300"; 
  $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7"; 
  $header[] = "Accept-Language: en-us,en;q=0.5"; 
  $header[] = "Pragma: "; // browsers keep this blank. 

  curl_setopt($curl, CURLOPT_URL, $url); 
  curl_setopt($curl, CURLOPT_USERAGENT, 'Googlebot/2.1 (+http://www.google.com/bot.html)'); 
  curl_setopt($curl, CURLOPT_HTTPHEADER, $header); 
  curl_setopt($curl, CURLOPT_REFERER, 'http://www.google.com'); 
  curl_setopt($curl, CURLOPT_ENCODING, 'gzip,deflate'); 
  curl_setopt($curl, CURLOPT_AUTOREFERER, true); 
  curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); 
  curl_setopt($curl, CURLOPT_TIMEOUT, 10); 

  $html = curl_exec($curl); // execute the curl command 
  curl_close($curl); // close the connection 

  return $html; // and finally, return $html 
} 

$response = disguise_curl('http://orcahub.com/unchecked-proxy-list/'); 
preg_match_all("/<a[\s]+[^>]*?href[\s]?=[\s\"\']+"."(.*?)[\"\']+.*?>"."([^<]+|.*?)?<\/a>/", $response, $matches, PREG_SET_ORDER );

foreach($matches as $value) { 
    var_dump($value);
};

+1!!! Sorry for the late reply but it works like a charm! I appreciate the help a ton! https://gyazo.com/223647f2cb5f954d223eb57733e535ab — John Butler, Dec 18 '15 at 16:23
Ok so, I ended up at another issue. It gives me a Server Error 500 when I use your script (Slightly Modified) to go into the .txt file. Please check my updated Post — John Butler, Dec 18 '15 at 16:50

Load All TXT Files In A External Directory

1 Answers1