Why is my PHP script stopping when generating thumbnails?

Question

I'm a PHP beginner and I'm trying to build a gallery page that will output thumbnails for all 195 images in a folder. These images average 8 MB each. Here's the current situation:

php.ini in the script folder

allow_url_fopen = On
display_errors = On
enable_dl = Off
file_uploads = On
max_execution_time = 999
max_input_time = 999
max_input_vars = 1000
memory_limit = 999M
post_max_size = 516M
session.gc_maxlifetime = 1440
session.save_path = "/var/cpanel/php/sessions/ea-php74"
upload_max_filesize = 512M
zlib.output_compression = Off

PHP / HTML code

<?php
DEFINE('UPLOAD_DIR', 'sources/');
DEFINE('THUMB_DIR', 'thumbs/');

function GenerateThumbnail($src, $dest)
{
    $Imagick = new Imagick($src);
    
    $bigWidth = $Imagick->getImageWidth();
    $bigHeight = $Imagick->getImageHeight();
    
    $scalingFactor = 230 / $bigWidth;
    $newheight = $bigHeight * $scalingFactor;
    
    $Imagick->thumbnailImage(230,$newheight,true,true);
    $Imagick->writeImage($dest);
    $Imagick->clear();
    
    return true;    
}

// Get list of files in upload dir
$arrImageFiles = scandir(UPLOAD_DIR);

// Remove non-images
$key = array_search('.', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

$key = array_search('..', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

$key = array_search('.ftpquota', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

$key = array_search('thumbs', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

?><!DOCTYPE HTML>
<html lang="fr">
<head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Select Image</title>
</head>
    <body>
<?php
foreach($arrImageFiles as $imageFile)
{
        $thumbFullPath =  THUMB_DIR . "th_" . $imageFile;
        $imageFullPath = UPLOAD_DIR . $imageFile;

        if (! file_exists($thumbFullPath))  
        {           
            GenerateThumbnail($imageFullPath, $thumbFullPath);
        }
        
        echo "<img alt='' src='" . $thumbFullPath . "'>";
    
}
?>              
    </body>
</html>

The two issues I don't know how to fix are:

the script seems to stop at some point before all thumbnails are generated (after 30-50 thumbs). No error is logged

the page is not loaded in the browser until the script stops and then I can see the (incomplete) output

Here's what I've already tried:

using output buffering (probably incorrectly because newbie)

concatenating the output in a variable and echoing it at the end (no change)

using different methods of generating the thumbnails (imagecopyresample, imagescale, ImageMagick, etc.) This only changes marginally the number of successful thumbnails.

Thanks for any ideas, I'm a bit lost.

Looks like a problem with the server timeout. How long does it take until the process stops? — David Pauli, Jun 03 '21 at 20:28
@DavidPauli -- Thanks for your comment. Load time varies quite a bit. I've tested the page load time and got: 48s, 123s, 57s, 115s. I'm puzzled! — Kerans, Jun 04 '21 at 12:28
*"php.ini in the script folder"* Why should that file have any effect? — Olivier, Jun 07 '21 at 19:09
@Olivier - Thanks for chipping in. As I said, I'm a newbie, and I understood (wrongly perhaps?) that placing a php.ini file there would allow me to modify some parameters (execution time, memory limit, etc.) Do you have any tips about that? I'm eager to learn. — Kerans, Jun 07 '21 at 23:12
@Kerans When PHP runs as CGI, you can put `.user.ini` files in directories to override settings (as explained [here](https://www.php.net/manual/en/configuration.file.per-user.php)). When PHP runs as an Apache module, only `.htaccess` files can be used. In any case, always call [`phpinfo()`](https://www.php.net/manual/en/function.phpinfo.php) to check the configuration. — Olivier, Jun 08 '21 at 07:16
@Olivier - wow, thanks, I didn't know that! I'll investigate using phpinfo, because it might just be that my settings have no effect hence the abrupt stop. Thanks! — Kerans, Jun 08 '21 at 14:59
As you said no error was logged, please check the error logs in `/var/log/httpd or apache2 or nginx or the folder you given in webserver)`. Please try to solve the issue based on the error you caught more than assumptions. *Note, try not to assign memory_limit to -1* — Rinshan Kolayil, Jun 14 '21 at 09:58

Kinglish · Answer 1 · 2021-06-12T08:51:20.860

The issue is definitely either a timeout or resource scarcity (memory or something like that). This article shows ImageMagick causing 315.03 MB memory usage per image and 47% CPU usage. That can compound/leak if you're cycling through images in the processor in a foreach loop. Image transformations can be expensive on server resources. It might also be a problem with one of your image files or some other oddity.

Ajax it

A solution to any of these problems is to process these in small batches. Create a simple ajax/php loop - using an html.php page for the ajax and a separate file for the processing, like this:

thumb-display.php

<script>
const batchSize = 5; // number of images per send
function doThumb(nextIndex) {
    $.ajax({
      url: "thumb-process.php",
      method: "post",
      data: {nextIndex: nextIndex, batchSize: batchSize},
      dataType: "json"
    }).done(function(response) {
      if (response.thumbs) {
         response.thumbs.forEach( th => $('.thumbs').append('<img alt="" src="'+th+'" />') ) ;
      }
      if (response.finished) alert('finished')
      else {
        console.log('processed batch index #' + (nextIndex-batchSize + ' - ' + nextIndex);
        nextIndex += batchSize;
        setTimeout(() => { doThumb(nextIndex) }, 1000);
      }
    }).error(function (e) {
       console.error('Error happened on batch index #' + (nextIndex-batchSize + ' - ' + nextIndex, ' - details: ', e);
    })
    ;
}

$(document).ready(() => { doThumb(0); } )
</script>


<div class='thumb-list'></div>

thumb-process.php

function GenerateThumbnail($src, $dest)
{
    $Imagick = new Imagick($src);
    $bigWidth = $Imagick->getImageWidth();
    $bigHeight = $Imagick->getImageHeight();
    $scalingFactor = 230 / $bigWidth;
    $newheight = $bigHeight * $scalingFactor;
    $Imagick->thumbnailImage(230,$newheight,true,true);
    $Imagick->writeImage($dest);
    $Imagick->clear();
    return true;    
}

// Get list of files in upload dir
$arrImageFiles = scandir(UPLOAD_DIR);

// Remove non-images
$key = array_search('.', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

$key = array_search('..', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

$key = array_search('.ftpquota', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);

$key = array_search('thumbs', $arrImageFiles);
if ($key !== false) 
    unset($arrImageFiles[$key]);
$nextIndex = $_POST['nextIndex'];
$max = min( ($nextIndex + $_POST['batchSize']), count($arrImageFiles)-1);
$thumbs = [];
while($nextIndex < $max){    
  $imageFile = $arrImageFiles[$nextIndex ];
  $thumbFullPath =  THUMB_DIR . "th_" . $imageFile;
  $imageFullPath = UPLOAD_DIR . $imageFile;
  if (!$imageFile) {
    $output = array('finished' => 1) ;
    die(json_encode($output));
  }
  if (! file_exists($thumbFullPath))  {           
    GenerateThumbnail($imageFullPath, $thumbFullPath);
  }
  $thumbs[]= $thumbFullPath ;
  $nextIndex++;
}

$output = array('thumbs' => $thumbs);
  
if ($nextIndex >= count($arrImageFiles)-1) $output['finished'] = 1 ;

echo json_encode($output);

.. and just sit back and watch your console. If you get a timeout or other error, you'll see the nextIndex it choked on, and you can reload your page with that number as a starting index instead of 0.

Of course you could gather all the file paths in an array in thumb-display.php and send each filepath through ajax (so as not to have to recount all files in that path each time), but personally I feel better about sending a number through post rather than an image path. Let me know if you'd rather have a big array of filepaths to send instead of an index#.

Thanks Kinglish, that sounds like a practical solution. What I don't get is -- why is there no output at all until the script stops? And how could I find what limit (memory or other) is reached? I have a cPanel, if that helps. Thanks for your time! — Kerans, Jun 07 '21 at 19:13
@Kerans - you could init error_reporting in your PHP file - https://stackoverflow.com/questions/1053424/how-do-i-get-php-errors-to-display -- I know cpanel used to have a way to view error logs as well. The benefit to my solution is that it's scalable - there could be 10,000 images to process and it would be fine. It takes a bit longer than a straight PHP loop, but is easier on your server and easier to troubleshoot — Kinglish, Jun 07 '21 at 19:17
@Kerans I did a little more research on ImageMagik (see new first paragraph) and also edited my script to allow for batches (rather than one by one) that you can set in javascript. cheers — Kinglish, Jun 12 '21 at 08:52

score 0 · Answer 2 · answered Jun 14 '21 at 17:35

@kinglish answer is good to load them via ajax, but I don't think it's the right approach. This alleviates some processing from the server as it doesn't run them all at once, but it's still not a good solution if you ever want many users accessing the 195 images

The biggest problem is here is that you are trying to process images on request, so every single page load will request and recreate all 195 images. This is bad for performance, and chances are 10 users on your site will crash your server just by refreshing the page a few times unless you're paying more than $10/mo for it.

So, here's a better solution.

Run your image processing script server side only on a loop of the 198 images. You can look up how to run a command to execute a php file, or just do "php yourfile.php" if you're SSH'd into a linux machine.

The script will run and process the images faster than when you do it via a browser request. [don't know the details, it's just that's how it works]

Store the images to a "thumbnail" directory and then just load them like you normally would via tags. TADA problem solved. You only process them once this way, the rest of the time you only serve them to the client. Yes, you now have both raw size and thumbnail versions, which feels wrong, but performs way way better than trying to regenerate them on every request.

Add some caching [load them to S3, or use cloudflare to cache them] and your site will be as fast as any.

Why is my PHP script stopping when generating thumbnails?

2 Answers2

Ajax it