Transfer a file of any type in 1k chunks over HTTP

Question

I need to transfer files of any type or size over HTTP/GET in ~1k chunks. The resulting file hash needs to match the source file. This needs to be done in native PHP without any special tools. I have a basic strategy but I'm getting odd results. This proof of concept just copies the file locally.

CODE

<?php

    $input="/home/lm1/Music/Ellise - Feeling Something Bad.mp3";
    $a=pathinfo($input);
    $output=$a["basename"];

    echo "\n> ".md5_file($input);
    $fp=fopen($input,'rb');

    if ($fp) {

        while(!feof($fp)) {
            $buffer=base64_encode(fread($fp,1024));
            // echo "\n\n".md5($buffer);
            write($output,$buffer);
        }
        fclose($fp);
        echo "\n> ".md5_file($output);
        echo "\n";
        
    }   
    
    function write($file,$buffer) { 
        // echo "\n".md5($buffer);
        $fp = fopen($file, 'ab');
        fwrite($fp, base64_decode($buffer));
        fclose($fp);    
    }   
    
?>

OUTPUT

> d31e102b1cae9c73bbf5a12615a8ea36
> 9f03f6c88ed61c07cb534922d6d31864

Thanks in advance.

nosurs · Accepted Answer · 2021-04-08T13:27:07.557

fread already advances the file pointer position, so there's no need to keep track of it. Same with frwite, so consecutive calls automatically append to the given file. Thus, you could simplify your approach to (code adapted from this answer on how to efficiently write a large input stream to a file):

$src  = "a.test";
$dest = "b.test";

$fp_src = fopen($src, 'rb');

if ($fp_src) {
    
    $fp_dest     = fopen($dest, 'wb');
    $buffer_size = 1024;

    while(!feof($fp_src)) {
        
        fwrite($fp_dest, fread($fp_src, $buffer_size));
    }

    fclose($fp_src);
    fclose($fp_dest);

    echo md5_file($src)."\n";  // 88e4af2f85080a280e7f00e50d96b7f7
    echo md5_file($dest)."\n"; // 88e4af2f85080a280e7f00e50d96b7f7
}

If you want to keep both processes separated, you'd do:

$src  = "a.test";
$dest = "b.test";

if (file_exists($dest)) {

    unlink($dest); // So we don't append to an existing file
}

$fp = fopen($src,'rb');

if ($fp) {

    while(!feof($fp)){

        $buffer = base64_encode(fread($fp, 1024));
        write($dest, $buffer);
    } 

    fclose($fp);
}

function write($file, $buffer) { 
    
    $fp = fopen($file, 'ab');

    fwrite($fp, base64_decode($buffer));
    fclose($fp);    
}   

echo md5_file($src)."\n";  // 88e4af2f85080a280e7f00e50d96b7f7
echo md5_file($dest)."\n"; // 88e4af2f85080a280e7f00e50d96b7f7

As for how to stream files over HTTP, you might want to have a look at:

Streaming a large file using PHP

I was able to improve the code dramatically based on your example, thanks. But in my case the read and write will occur on different systems. When I separate read/write and base64 the buffers, they seem to match during but the resulting file is much larger and of course has a different hash. — pQL7CmRP2i4, Apr 08 '21 at 12:39
I've extended my answer a bit, so my second example should be pretty close to your code now - the only thing I really changed was to check if the destination file exists & delete it if it does. I totally missed that at first, maybe you did, too? — nosurs, Apr 08 '21 at 13:20

Transfer a file of any type in 1k chunks over HTTP

1 Answers1