PHP shell_exec returns empty string after a while

Question

I made a Symfony console command which uses pngquant to process and compress a long list of images. The images are read from a CSV file.

The batch is working fine until the end in local environment but in the stage environment it works for about 5 minutes and then it starts returning empty result from the shell_exec command. I even made a retry system but it's always returning empty result:

// escapeshellarg() makes this safe to use with any path
// errors are redirected to standard output
$command = sprintf(
    '%s --quality %d-%d --output %s --force %s 2>&1',
    $this->pngquantBinary,
    $minQuality,
    $maxQuality,
    escapeshellarg($tempPath),
    $path
);

// tries a few times
$data    = null;
$attempt = 0;

do {
    // command result
    $data = shell_exec($command);

    // error
    if (null !== $data) {
        $this->logger->warning('An error occurred while compressing the image with pngquant', [
            'command' => $command,
            'output'  => $data,
            'cpu'     => sys_getloadavg(),
            'attempt' => $attempt + 1,
            'sleep'   => self::SLEEP_BETWEEN_ATTEMPTS,
        ]);
        sleep(self::SLEEP_BETWEEN_ATTEMPTS);
    }

    ++$attempt;
} while ($attempt < self::MAX_NUMBER_OF_ATTEMPTS && null !== $data);

// verifies that the command has finished successfully
if (null !== $data) {
    throw new \Exception(sprintf('There was an error compressing the file with command "%s": %s.', $command, $data));
}

The problem is that the same command executed in another shell in the same environment works fine! I mean, when I log the error, if I put the exactly same command in another instance on the same server, works fine.

Even from the Symfony logs I can't read any error, where should I look for a more detailed error?

What can be causing this? Memory and processor are fine during execution!

`if (null !== $data)` shouldn't it be `if (null === $data)` ? — Cid, Mar 03 '20 at 08:36
I am redirecting errors to the standard output so the command returns `null` if it runs correctly (the pngquant command has no output). — StockBreak, Mar 03 '20 at 08:38
What confuses me about the question is that you say `and then it starts returning empty result from the shell_exec command.`... but a "non empty result" is an error condition in your code, so "empty result" is expected, right? — yivi, Mar 03 '20 at 08:43
Empty result as an empty string is an error but `null` is not (in my code). The empty result should actually contain the error message but the system is not returning it. — StockBreak, Mar 03 '20 at 09:00
instead of using ```shell_exec``` use ```exec```, Ex: exec(command,$output), print $output and check. — HP371, Mar 03 '20 at 09:05
@HP371 with `exec` the command starts returning -1 (as `$return_var`) and empty array (as `$output`) after a while. It works well for about 5 minutes. — StockBreak, Mar 03 '20 at 09:34

StockBreak · Accepted Answer · 2020-03-06T16:30:55.083

After many attempts I read this question:

Symfony2 Process component - unable to create pipe and launch a new process

The solution was to add a call to gc_collect_cycles after flush during the loop!

if ($flush || 0 === ($index % self::BATCH_SIZE)) {
    $this->em->flush();
    $this->em->clear();

    // clears the temp directory after flushing
    foreach ($this->tempImages as $tempImage) {
        unlink($tempImage);
    }
    $this->tempImages = [];

    // forces collection of any existing garbage cycles
    gc_collect_cycles();
}

IMPORTANT: also keep an eye on the number of disk inodes.

df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            125193     392 124801    1% /dev
tmpfs           127004     964 126040    1% /run
/dev/vda2      5886720 5831604  55116  100% /

PHP shell_exec returns empty string after a while

1 Answers1