0

I've got a cron job that scans URLs to see if they're online or not. My problem is it never completes the 844 websites. It will vary between 260 to a little over 300.

The cron job calls this PHP file every 30 minutes, but it never completes my full list. Is there anything I'm not doing correctly to prevent the loop from completing the task?

// php file
//first part gets the csv file and reads the 844 sites

if (($handle = fopen("/public/csvs/" . $csv, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
        $num = count($data);

        for ($c = 0; $c < $num; $c++) {

           $site = $data[$c];

                   $curl = curl_init();
                    curl_setopt_array( $curl, array(
                        CURLOPT_HEADER => true,
                        CURLOPT_NOBODY => true,
                        CURLOPT_RETURNTRANSFER => true,
                        CURLOPT_URL => 'https://'.$site ) );
                    $headers = explode( "\n", curl_exec( $curl ) );
                    curl_close( $curl );

                     // gets the status of the URL then 
                     fills it in the database.

                    $statushttp = $headers[0];



                    $mysqltime = date("Y-m-d H:i:s", $phptime);


                        //$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
                        //$sql = $conn->prepare($sql);
                        $sql = "INSERT INTO csv (http_status,status,url,csv,related)
                     VALUES ('$statushttp','$status','$site','$csv',1)";

                        // use exec() because no results are returned

                        $conn->exec($sql);
                        //echo $site ." ".$statushttp."<br>";
                        //echo $statusCode."<br>";



        }

    }

}
Sayed Mohd Ali
  • 2,156
  • 3
  • 12
  • 28
Drew Peer
  • 377
  • 2
  • 14
  • 1
    You need to increase the execution time of your PHP. https://stackoverflow.com/questions/3829403/how-to-increase-the-execution-timeout-in-php – Sayed Mohd Ali Dec 24 '18 at 07:00
  • 1
    CLI is not (should not be) subject to the same time limits as HTTP. Hard to say with no details on the hosting. – ArtisticPhoenix Dec 24 '18 at 07:10
  • I did increase the max execution time, but like ArtisticPhoenix pointed out it doesn't effect it with CLI. I tried set_time_limit(0); at the top. – Drew Peer Dec 24 '18 at 07:47
  • Try to increase the cron job interval to 2 hours to make sure it completes. If we average 5secs per website it will finish at 1hr and 10mins for 844 sites. – McBern Dec 24 '18 at 08:26

1 Answers1

0

Tried this on a different server and it worked no problems.

Drew Peer
  • 377
  • 2
  • 14