2

I am developing a File Uploader for big file. Upload from HTML script and send by byte from Javascript using ArrayBuffer and Unit8Array to PHP. The PHP script will stream the file and save it into folder.

Here's my Javascript looks like

function upload(fileInputId, fileIndex)
    {
        var file = document.getElementById(fileInputId).files[fileIndex];
        var blob;
        var reader = new FileReader();
        reader.readAsBinaryString(file); 
        reader.onloadend  = function(evt)
        {
            xhr = new XMLHttpRequest();

            xhr.open("POST", 'upload.php?name=' + file.name, true);

            XMLHttpRequest.prototype.mySendAsBinary = function(text){
                var data = new ArrayBuffer(text.length);
                var ui8a = new Uint8Array(data, 0);
                for (var i = 0; i < text.length; i++){ 
                    ui8a[i] = (text.charCodeAt(i) & 0xff);

                }

                if(typeof window.Blob == "function")
                {
                     blob = new Blob([data]);
                }else{
                     var bb = new (window.MozBlobBuilder || window.WebKitBlobBuilder || window.BlobBuilder)();
                     bb.append(data);
                     blob = bb.getBlob();
                }

                this.send(blob);
            }

            var eventSource = xhr.upload || xhr;
            eventSource.addEventListener("progress", function(e) {
                var position = e.position || e.loaded;
                var total = e.totalSize || e.total;
                var percentage = Math.round((position/total)*100);
            });

            xhr.onreadystatechange = function()
            {
                if(xhr.readyState == 4)
                {
                    if(xhr.status == 200)
                    {
                        console.log("Done");
                    }else{
                        console.log("Fail");
                    }
                }


            };

            xhr.mySendAsBinary(evt.target.result);
        };
    }

This is my upload.php

$inputHandler = fopen('php://input', "r");
$loc = "uploads/" . $_GET["name"];
$fileHandler = fopen($loc, "w+");

while(true) {
    $buffer = fgets($inputHandler, 4096);



    if (strlen($buffer) == 0) {
        fclose($inputHandler);
        fclose($fileHandler);
        return true;
    }

    fwrite($fileHandler, $buffer);
}

My question is, how do I encrypt those upload file using AES or mcrypt while the file is in streaming mode?

Mr Hery
  • 829
  • 1
  • 7
  • 25
  • 1
    Why would you want to, I sort of thought that was what `https` SSL/TLS was for. Also depending how big it is, AES will eventually barf on you. Maybe 50-100MB. You can do something like I did. Take about 1MB, encrypt it, add a `:` take another 1MB encrypt it. etc. Then when you decrypt, you just read the file tell you get to a `:` decode a chunk, etc. etc. Basically encrypt it one chunk at a time. – ArtisticPhoenix Mar 30 '18 at 02:00
  • I mean is, the file will stored as encrypted file. maybe looks like `image.enc` or something like that – Mr Hery Mar 30 '18 at 02:03
  • How big is it? You will hit a memory wall, sooner or later trying to encrypt a large file. I forget how big we got up to, but we have 54GB of ram and had to encrypt chunk by chunk. – ArtisticPhoenix Mar 30 '18 at 02:04
  • I would also avoid `mycrypt - This feature was DEPRECATED in PHP 7.1.0, and REMOVED in PHP 7.2.0.` I just had to re-code my AES, because were moving to PHP7 in a few months. And the encryption is not backwards compatible, meaning I had to re-ecrypt everything. So I decided to put the big file runs out of memory issue to bed. – ArtisticPhoenix Mar 30 '18 at 02:06
  • On our system we put maximum as 1GB only. Encrypt by MB make sense – Mr Hery Mar 30 '18 at 02:08
  • So basically in your case, you will let the buffer fill to a set lenght. Maybe 5 or 10 MB. Then use something like `open_ssl` or `PHPSecLib` to encrypt the chunk, then base64 encode it, add a separator, write it to disc, grab the next chunk... etc.. There are plenty of examples on how to do AES, just stay away from the ones that use mycrypt like I mentioned. In my case I actually wrote it into a file stream handler, where I give it an input stream and an output stream and a key using PHPSecLib.... – ArtisticPhoenix Mar 30 '18 at 02:11
  • Last thing is I would share the code with you, but I don't have it on me, and I would have to rewrite like 30% of it as I am not allowed to share work code... :( – ArtisticPhoenix Mar 30 '18 at 02:16
  • Thanks for your info. If you could put some example I would really love to. – Mr Hery Mar 30 '18 at 02:18
  • Ok, give me a few minutes. – ArtisticPhoenix Mar 30 '18 at 02:30

1 Answers1

3

It was something like this. This is from memory and untested, because I don't have the PHPSecLib library on my Laptop, and I am too lazy to set that all up...

require __DIR__ . '/vendor/autoload.php';

use phpseclib\Crypt\AES;
use phpseclib\Crypt\Random;

AESStreamEncode($input, $output, $key)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);
    
    $iv = Random::string($cipher->getBlockLength() >> 3);
    $cipher->setIV($iv);
    
    $base64_iv = rtrim(base64_encode($iv), '='); //22 chars
    
    fwrite($output, $base64_iv); //store the IV this is like a salt

    while(!feof($input)) {
        $contents = fread($input, 1000000); //number of bytes to encrypt 
        $encrypted = $cipher->encrypt($contents);
        //trim the = or ==, and replace with :, write to output stream.
        fwrite($output, rtrim(base64_encode($encrypted), '=').':'); 
    }
}

AESStreamDecode($input, $output, $key)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);
    
    $buffer = '';
    $iv = false;
    
    while(!feof($input)) {
        $char = fgetc($input); //get a single char
        if($char ==':'){
            if(!$iv){
                $iv = base64_decode(substr($buffer, 0, 22).'=');  //iv is the first 22 of the first chunk.
                $cipher->setIV($iv);
                $buffer = substr($buffer, 22); //remove the iv
            }
            $buffer = base64_decode($buffer.'='); //decode base64 to bin
            $decrypted = $cipher->decrypt($buffer);
            fwrite($output, $decrypted);
            
            $buffer = ''; //clear buffer.
        }else{
            $buffer .= $char;
        }
    }
}

Where $input and $output are valid resource stream handles like from fopen etc.

 $input = fopen($filepath, 'r');
 $output = fopen($ohter_filepath, 'w');

 AESStreamEncode($input, $output, $key);

This lets you use things like php://output as the stream if downloading the decrypted file.

You have to remove the = because it is sometimes missing or 2 of them, So we cant rely on them as a separator. I usually just put 1 back on and it always decodes it correctly. I think it's just some padding anyway.

References

PHPSecLib on GitHub

PHPSecLib Examples

The encrypted file should look something like this:

xUg8L3AatsbvsGUaHLg6uYUDIpqv0xnZsimumv7j:zBzWUn3xqBt+k1XP0KmWoU8lyfFh1ege:nJzxnYF51VeMRZEeQDRl8:

But with longer chunks. The IV is like a salt and it's pretty common practice to just add it to the front or back of the encrypted string. So for example

[xUg8L3AatsbvsGU]aHLg6uYUDIpqv0xnZsimumv7j:

The part in the [] is the IV, (its 22 chars long after base64_encode) I counted it many times and it always comes out that long. We only need to record the IV and set it one time. I suppose you could do a different IV for each chunk, but whatever.

If you do use PHPSecLib, it also has some nice sFTP stuff in it. Just make sure to get the 2.0 version. Basically it has some fallbacks and native PHP implementations for different encryption algos. So like it would try open_ssl then if you were missing it, it would use their native implementation. I use it for sFTP, so I already had it available. sFTP requires an extension ssh2_sftp and If I recall it was only available on Linux at the time we set things up.

UPDATE

For downloading you can just issue the headers then give the decode function the output stream, something like this

 $input = fopen('encrypted_file.txt', 'r');
 $output = fopen('php://output', 'w');

 header('Content-Type: "text/plain"');
 header('Content-Disposition: attachment; filename="decoded.txt"');

 header('Expires: 0');
 header('Cache-Control: must-revalidate, post-check=0, pre-check=0, max-age=0');
 header("Content-Transfer-Encoding: binary");
 header('Pragma: public');

 //header('Content-Length: '.$fileSize);  //unknown

 AESStreamDecode($input, $output, $key);

These are pretty standard headers. The only real catch is because the filesize is different when it's encryped you can't just simply get the size of the file and use that as it will be quite a bit bigger. Not passing the filesize won't prevent the download, it just wont have an estimated time etc.

But because we know the size before encrypting it, we could embed it in the file data itself like this:

 3555543|xUg8L3AatsbvsGUaHLg6uYUDIpqv0xnZsimumv7j:zBzWUn3xqBt+k1XP0KmWoU8lyfFh1ege:nJzxnYF51VeMRZEeQDRl8:

And then pull it out when we do the download, but you would have to use as separate function to get it and it might be a bit tricky to not mess up decoding the file.

Honestly I think it's more hassle then it's worth.

UPDATE2

Anyway, I worked up these changes for embedding the file size, it's an option, but it could also mess up the decryption of the file if not done carefully. (I haven't tested this)

AESStreamEncode($input, $output, $key, $filesize = false)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);

    $iv = Random::string($cipher->getBlockLength() >> 3);
    $cipher->setIV($iv);

    $base64_iv = rtrim(base64_encode($iv), '='); //22 chars
    
    //Option1 - optional filesize
    if(false !== $filesize){
        //add filesize if given in the arguments
        fwrite($output, $filesize.'|');
    }
    
    /*
        //Option2: using fstat, remove '$filesize = false' from the arguments
        $stat = fstat($input);
        fwrite($output, $stat['size'].'|');
    */

    fwrite($output, $base64_iv); //store the IV this is like a salt

    while(!feof($input)) {
        $contents = fread($input, 1000000); //number of bytes to encrypt 
        $encrypted = $cipher->encrypt($contents);
        //trim the = or ==, and replace with :, write to output stream.
        fwrite($output, rtrim(base64_encode($encrypted), '=').':'); 
    }
}

So now we should have the filesize 3045345|asdaeASE:AEREA etc. Then we can pull it back out when decrypting.

AESStreamDecode($input, $output, $key)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);

    $buffer = '';
    $iv = false;
    $filesize = null;

    while(!feof($input)) {
        $char = fgetc($input); //get a single char
        if($char =='|'){
            /*
              get the filesize from the file,
              this is a fallback method, so it wont affect the file if
              we don't pull it out with the other function (see below)
            */
            $filesize = $buffer;
            $buffer = '';
        }elseif($char ==':'){
            if(!$iv){
                $iv = base64_decode(substr($buffer, 0, 22).'=');  //iv is the first 22 of the first chunk.
                $cipher->setIV($iv);
                $buffer = substr($buffer, 22); //remove the iv
            }
            $buffer = base64_decode($buffer.'='); //decode base64 to bin
            $decrypted = $cipher->decrypt($buffer);
            fwrite($output, $decrypted);

            $buffer = ''; //clear buffer.
        }else{
            $buffer .= $char;
        }
    }
    //when we do a download we don't want to wait for this
    return $filesize;
}

The decode get filesize part acts as a fallback, or if you don't need it then you don't have to worry about it messing the file up when decoding it. When downloading we can use the following function, that way we don't have to wait for the file to be completely read to get the size (this is basically the same as what we did above).

//We have to use a separate function because
//we can't wait tell reading is complete to 
//return the filesize, it defeats the purpose
AESStreamGetSize($input){
    $buffer = '';
    //PHP_INT_MAX (maximum allowed integer) is 19 chars long
    //so by putting a limit of 20 in we can short cut reading
    //if we can't find the filesize
    $limit = 20;
    $i; //simple counter.
    while(!feof($input)) {
        $char = fgetc($input); //get a single char
        if($char =='|'){
            return $buffer;
        }elseif($i >= $limit){
            break;
        }
        $buffer .= $char;
        ++$i; //increment how many chars we have read
    }
    return false;
}

Then when downloading you just need to make a few changes.

$input = fopen('encrypted_file.txt', 'r');
//output streams dumps it directly to output, lets us handle larger files
$output = fopen('php://output', 'w');
//other headers go here

if(false !== ($filesize = AESStreamGetSize($input))){
    header('Content-Length: '.$fileSize);  //unknown
    //because it's a file pointer we can take advantage of that
    //and the decode function will start where the getSize left off.
    // or you could rewind it because of the fallback we have.
    AESStreamDecode($input, $output, $key);
}else{
    //if we can't find the filesize, then we can fallback to download without it
    //in this case we need to rewind the file
    rewind($input);
    AESStreamDecode($input, $output, $key);
}

If you want to shorten this you can just do it this way too, it's only about 19 chars at most so it's not to big a performance issue.

 if(false !== ($filesize = AESStreamGetSize($input))) header('Content-Length: '.$fileSize);

 rewind($input);
 AESStreamDecode($input, $output, $key);

Basically above, we just do the filesize header (or not) and then rewind and do the download. It will re-read the filesize, but that's pretty trivial.

For reference fstat(), Hopefully that makes sense.

Community
  • 1
  • 1
ArtisticPhoenix
  • 21,464
  • 2
  • 24
  • 38
  • Thanks for the code. Btw, the `AESStreamEncode` should be in the `while(true)` code block or the outside? The input is the `$inputHandler` (which is I am using php://input) or `$fileHandler` (wrtten file)? – Mr Hery Mar 30 '18 at 03:03
  • I've never done an upload like in your Question, so I don't know if it gives you the whole file or just some bytes at a time. This is just from memory ... lol. I wrote that like 2 weeks ago or so. I also didn't test is so hopefully there are not many errors. – ArtisticPhoenix Mar 30 '18 at 03:07
  • I think in your case, I would replace the read loop `while(!feof)` with your `while(true)` then `$input = $inputHandler` and `$output = $fileHandler`, You also may have to loop a few times to get the amount of bytes you need, as I said I don't know if that is like an asynchronous stream, where you can only get so many bytes at a time. `fgets` reads only 1 line for example. – ArtisticPhoenix Mar 30 '18 at 03:14
  • It may be possible to read your stream just like a normal file. As I mentioned above `fgets` only reads to the next `\n` new line or the byte limit. So you could check the length of buffer if you need to loop more then once to get the number of bytes you want. That is kind of a balance, you don't want it so long that it causes memory issues, but you also don't want it so short you call `encrypt` to many times. – ArtisticPhoenix Mar 30 '18 at 03:33
  • The size of encrypted file are bigger 30-40% than original file. Is it normal? – Mr Hery Apr 08 '18 at 02:06
  • Yes, because when encrypting and base64 encoding you are reducing the character set, so some characters are represented (in base64) by more then one character. Encrypting comes out in binary and it cant be safely stored in a text file without corrupting it as I understand it, so you have to do something like base64 encoding. Therefore you wind up with more characters because of that. – ArtisticPhoenix Apr 08 '18 at 02:09
  • You can easily test this by using base64 on an ordinary string and counting. So this site https://www.base64encode.org/ can do it online, if I put `hello` in I get `SGVsbG8=`, `hello` is 6 chars, and the other is `7` without the `=`, so multiply that by thousands of words, and that's just the base64 part. – ArtisticPhoenix Apr 08 '18 at 02:13
  • How I can decrypt the file and stream to download? Like I put `header("Content-Type: image/jpeg");` and `header("Content-Dsposition: attachnment; filename: 't.hjpg'");` then `echo $decsrypted;` something like this. – Mr Hery Apr 13 '18 at 03:21
  • Nice one sir :). All cleared up and good to go. The header content type i put it empty because I dont save the mime. I only put the header filename it all goes well. Thank you sir – Mr Hery Apr 13 '18 at 05:03