Why is exec() silently discarding bytes?

Question

After a lot of struggling, I found out some behavior I can't explain / solve, so asking here for help. On our server (Ubuntu 16.04.5 LTS with PHP 7.0.30), we're using some tools outside of the 'httpdocs', which are called using exec() to get their outputs. In this case, it's a QRCode-generator.

Some of the QR-codes however, won't be displayed. We got an output from the tool (outputting data of a PNG), but when we show it as an image, it appears to be broken for some reason.

After a lot of debugging I found out the result sometimes differs 1 or 2 bytes from the output of the tool.

I did my last debugging using the QR-code below (12345), which is a file of 240 bytes. However, when outputting it eventually, the length appears to be 239 bytes, so we lost a byte somewhere?

I found out using exec(), we're creating an array of the output. Trailing whitespace, such as \n, is not included in this array, so implode() to glue the array back to a string, like below:

<?php

$cmd = 'cat qr.png';
$output = array();
$exit_code = 0;

exec($cmd, $output, $exit_code);

if($exit_code === 0)
{
    header ("Content-type: image/png;");
    print implode(PHP_EOL, $output);
}
die();

But for some reason, a byte is lost in this process? I already tried some other solutions using shell_exec() (but no return_var here, so we can't check validate the proces...)

<?php

$command = 'cat qr.png';

$output = shell_exec($command);
if($output !== null)
{
    header ("Content-type: image/png;");
    print $output;
}
die();

... and using passthru() (but this outputs the contents directly, which isn't desired. Output-buffering like in the example below isn't possible in the actual code...)

<?php

$command = 'cat qr.png';
$return_var = 0;

ob_start();
passthru($command, $return_var);

if($return_var === 0)
{
    header ("Content-type: image/png;");
    $output = ob_get_clean();
    print $output;
}
die();

So far, the exec()-function always worked pretty well for us, resulting in a $return_var AND $output, but now I'm losing bytes. I already tried some variants of the PHP_EOL, we're now using as glue (\n, \r and \n\r), but with the first 2 line-endings, I still only got 239 bytes and with the last one, I end up with 241 bytes, so 1 byte to much.

Why am I losing a byte here? What is the correct way to convert the $output-array back into a string? Is there a way to get the 240-bytes output back by imploding the array? Or are there other functions I haven't found yet to execute a command which will give me the output as well as the return_var?

Answer 1

According to the manual entry for exec, the function returns the last line of output. Whether that last line is also included as the last element of $output is unclear.

But more importantly the the manual states:

Trailing whitespace, such as \n, is not included in this array.

I would guess that this is per-element of the array (i.e. per-line of your output), but either way, your whitespace will be significant here, because you're not dealing with text - all the bytes are important and meaningful here. Remember, whitespace doesn't just mean \n. It can also mean \t or (a space), for example.

In the event that a line ends with F\t\n (a capital F, or any other non-whitespace character, a tab, then a newline), both whitespace characters would be stripped from the end. When doing your implode operation, you might be putting the \n back, but you'll never know about the \t that was stripped.

The key thing to realise here is that exec is expecting to be dealing with plain text not raw binary data.

I would suggest that instead of using cat qr.png, you use base64 qr.png to encode the binary data into an ASCII string, then decode it in PHP with base64_decode. If you're not going to be using cat in reality, you can still pipe the output of your QR-generating command through base64 like this: generate-qr-png |base64. There shouldn't be any significant whitespace in that situation, so nothing would get munged by exec.