I have a script running for a Laravel 5.4 webapplication that is supposed to download a big amount of images (10k). I'm wondering what the best way to handle this would be. I currently grab the base64_encode()
data from the remote image and write it to a local folder with the function file_put_contents()
. This works fine but some images can take more than 10 seconds to download/write, image that times a 10 thousand. Fair enough these images are rather big but I would like to see this process happen faster and thus I am asking for advice!
My current process is like this;
- I read a JSON file containing all the image links I have to download.
- I convert the JSON data to an array with
json_decode()
and I loop through all the links with aforeach()
loop and let curl handle the rest.
All the relevant parts of the code look like this:
<?php
// Defining the paths for easy access.
$__filePath = public_path() . DIRECTORY_SEPARATOR . "importImages" . DIRECTORY_SEPARATOR . "images" . DIRECTORY_SEPARATOR . "downloadList.json";
$__imagePath = public_path() . DIRECTORY_SEPARATOR . "importImages" . DIRECTORY_SEPARATOR . "images";
// Decode the json array into an array readable by PHP.
$this->imagesToDownloadList = json_decode(file_get_contents($__filePath));
// Let's loop through the image list and try to download
// all of the images that are present within the array.
foreach ($this->imagesToDownloadList as $IAN => $imageData) {
$__imageGetContents = $this->curl_get_contents($imageData->url);
$__imageBase64 = ($__imageGetContents) ? base64_encode($__imageGetContents) : false;
if( !file_put_contents($__imagePath . DIRECTORY_SEPARATOR . $imageData->filename, base64_decode($__imageBase64)) ) {
return false;
}
return true;
}
And the curl_get_contents functions looks like this:
<?php
private function curl_get_contents($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
I hope someone could englighten me with possible improvements that I could apply on the current way I'm handling this mass-download.