0

I have a script running for a Laravel 5.4 webapplication that is supposed to download a big amount of images (10k). I'm wondering what the best way to handle this would be. I currently grab the base64_encode() data from the remote image and write it to a local folder with the function file_put_contents(). This works fine but some images can take more than 10 seconds to download/write, image that times a 10 thousand. Fair enough these images are rather big but I would like to see this process happen faster and thus I am asking for advice!

My current process is like this;

  1. I read a JSON file containing all the image links I have to download.
  2. I convert the JSON data to an array with json_decode() and I loop through all the links with a foreach() loop and let curl handle the rest.

All the relevant parts of the code look like this:

<?php
// Defining the paths for easy access.
$__filePath     = public_path() . DIRECTORY_SEPARATOR . "importImages" . DIRECTORY_SEPARATOR . "images" . DIRECTORY_SEPARATOR . "downloadList.json";
$__imagePath    = public_path() . DIRECTORY_SEPARATOR . "importImages" . DIRECTORY_SEPARATOR . "images";

// Decode the json array into an array readable by PHP.
$this->imagesToDownloadList = json_decode(file_get_contents($__filePath));

// Let's loop through the image list and try to download
// all of the images that are present within the array.
foreach ($this->imagesToDownloadList as $IAN => $imageData) {
    $__imageGetContents = $this->curl_get_contents($imageData->url);
    $__imageBase64 =  ($__imageGetContents) ? base64_encode($__imageGetContents) : false;

    if( !file_put_contents($__imagePath . DIRECTORY_SEPARATOR . $imageData->filename, base64_decode($__imageBase64)) ) {
        return false;
    }

    return true;
}

And the curl_get_contents functions looks like this:

<?php
private function curl_get_contents($url)
{
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
    $data = curl_exec($ch);
    curl_close($ch);

    return $data;
}

I hope someone could englighten me with possible improvements that I could apply on the current way I'm handling this mass-download.

CarrotCake
  • 19
  • 7
  • Bulk download - have a look here: _Allows the processing of multiple cURL handles asynchronously_ http://php.net/manual/en/function.curl-multi-init.php – JustOnUnderMillions Jan 31 '17 at 14:29
  • Possible duplicate of [PHP Multiple Curl Requests](http://stackoverflow.com/questions/3900153/php-multiple-curl-requests) – online Thomas Jan 31 '17 at 14:29
  • And write the files with cUrl http://stackoverflow.com/questions/25081667/curlopt-file-curl-multi-exec-and-fclose/36390293 – JustOnUnderMillions Jan 31 '17 at 14:32
  • Ah I haven't stumbled upon that function while searching around google, strangely enough.. I'm going to implement it and run it through some time tests to see the difference it makes. Thanks for the quick replies! – CarrotCake Jan 31 '17 at 14:49

0 Answers0