Effecient way for encrypt large data using Crypto++

Question

I am looking for a solution to encrypt/decrypt a big file (some GBs) using Crypto++. Several days ago, I was thinking about using StringSource in this question, read the whole file into a char array, then encrypt/decrypt it using CTR mode.

Encrypt a string to a string

string  encryptString(string plain, byte key[], int sizeKey, byte iv[], int sizeIV){
    string cipher;
    try{
        CTR_Mode< AES >::Encryption e;
        e.SetKeyWithIV(key, sizeKey, iv, sizeIV);

        StringSource s(plain, true,
            new StreamTransformationFilter(e,
            new StringSink(cipher)
            )
            );

#if 0
        StreamTransformationFilter filter(e);
        filter.Put((const byte*)plain.data(), plain.size());
        filter.MessageEnd();

        const size_t ret = filter.MaxRetrievable();
        cipher.resize(ret);
        filter.Get((byte*)cipher.data(), cipher.size());
#endif
        return cipher;
    }
    catch (const CryptoPP::Exception& e)
    {
        cerr << e.what() << endl;
        return NULL;
    }
}

Decrypt a string to a string

string  decryptString(string cipher, byte key[], int sizeKey, byte iv[], int sizeIV){
    string reco;
    try{
        CTR_Mode< AES >::Decryption d;
        d.SetKeyWithIV(key, sizeKey, iv, sizeIV);

        StringSource s(cipher, true,
            new StreamTransformationFilter(d,
            new StringSink(reco)
            )
            );

#if 0
        StreamTransformationFilter filter(e);
        filter.Put((const byte*)plain.data(), plain.size());
        filter.MessageEnd();

        const size_t ret = filter.MaxRetrievable();
        cipher.resize(ret);
        filter.Get((byte*)cipher.data(), cipher.size());
#endif
        return reco;
    }
    catch (const CryptoPP::Exception& e)
    {
        cerr << e.what() << endl;
        return reco;
    }
}

Wrap the encrypt string above

char* encrypt(char * plainText, byte key[], int sizeKey, byte iv[], int sizeIV, long len){
    string ss(plainText, len);
    long lengSS = ss.length();

    string cipher = encryptString(ss, key, sizeKey, iv, sizeIV);
    len = cipher.size();
    char * writable = new  char[len];
    std::copy(cipher.begin(), cipher.end(), writable);
    writable[len] = '\0';
    return writable;
}

Wrap the decrypt string above

char* decrypt(char * cipher, byte key[], int sizeKey, byte iv[], int sizeIV, long len){
    string ss(cipher, len);
    long lengSS = ss.length();
    string recovered = decryptString(ss, key, sizeKey, iv, sizeIV);
    //FileUtil::writeStringToFile("d2.txt", recovered);
    char * writable = new char[recovered.size() ];
    std::copy(recovered.begin(), recovered.end(), writable);
    writable[recovered.size() ] = '\0';
    return writable;
}

The main

int main(int argc, char* argv[])
{
    AutoSeededRandomPool prng;

    byte key[AES::DEFAULT_KEYLENGTH] = { '1', '2', '3', '4', '5', '6', '7', '8', '1', '2', '3', '4', '5', '6', '7', '8' };
    byte iv[AES::BLOCKSIZE] = { '8', '7', '6', '5', '4', '3', '2', '1', '8', '7', '6', '5', '4', '3', '2', '1' };
    //prng.GenerateBlock(iv, sizeof(iv));
    char* pdata = "CTR encrypt mode 1234567890";
    char * sourceFileName = "1MB.txt";
    char * targetFileName = "1MB.aes.crpp.txt";

    /*
    Looking for parameters.
    param 0: e to encrypt. d to decrypt
    param 1: sourceFileName
    param 2: targetFileName
    */
    if (argc > 1){

        char * action = argv[1];
        sourceFileName = argv[2];
        targetFileName = argv[3];
        long size = 0;

        char * pdata = FileUtil::readAllByte(sourceFileName, size);

        string ext = getExtension(sourceFileName);

        char * result1 = NULL;
        if (strcmp(action, "e") == 0){
            result1 = encrypt(pdata, key, sizeof(key), iv, sizeof(iv), size);
        }
        else if (strcmp(action, "d") == 0){
            result1 = decrypt(pdata, key, sizeof(key), iv, sizeof(iv), size);
        }
        FileUtil::writeFile(targetFileName, result1, size);
        //FileUtil::writeStringToFile(targetFileName, recovered1);
    }
    else{
        cout << "Missing/Invalid params" << endl;

    }
    return 0;

}

Now this solution can encrypt/decrypt file have size up to several hundreds MB. But when I using it with 1GB text file, its got error:

Unhandled exception at 0x74DFA6F2 in AES-CRPP.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x00AFE480.

at the line:

char*encrypt(...){
string ss(data, len)
...
}

I have 8GB of RAM and its look like reading 1GB of char and convert it into string cause the error. In another question, jww gave me a hint at the bottom, but I cannot make it work. Is there a solution for large byte (char) array encrypt using Crypto++

About using FileSource Document here: enter link description here Code example:

EAX< Blowfish >::Encryption e1;
        e1.SetKeyWithIV(key, key.size(), iv, sizeof(iv));

        CryptoPP::FileSource fs1(ofilename.c_str(), true,
            new AuthenticatedEncryptionFilter(e1,
            new FileSink(efilename.c_str())
            ));



        EAX< Blowfish >::Decryption d2;
        d2.SetKeyWithIV(key, key.size(), iv, sizeof(iv));

        CryptoPP::FileSource fs2(efilename.c_str(), true,
            new AuthenticatedDecryptionFilter(d2,
            new FileSink(rfilename.c_str()),
            AuthenticatedDecryptionFilter::THROW_EXCEPTION
            ));

It doesn't allow me to export the output cipher to a temporatory char * variable, instead of saving directly to a file. Of course I can wait for the file to file encryption to completed, then read the output file into the memory again, but its seem so bad.

Answer 1

As you told you want to encrypt big file while consuming less memory in the RAM. So you can use pipeline used in the crypto++. Please refer to this link https://www.cryptopp.com/wiki/Pumping_Data

Now here you will only see how to limit the size of your process. So if you want to do the encryption of big file. It is always preferable to encrypt big file in block by block. So you can use cipher block techniques of crypto++.

You can use StreamTransformationFilter as it automatically encrypt the file in blocks.

I have reproduce the code to do the same. The following code will encrypt the big size file using pipeline process. It won't take more than 10MB of RAM in your computer.

bool Widget::optimize_enc(const char *filename){
    try
        {
            string pwd="qwertyuiopasdfgh";
            const char* enc="out.aes";

            byte iv[ CryptoPP::CIPHER::BLOCKSIZE ];
            memset( iv, 0x01, CryptoPP::CIPHER::BLOCKSIZE );

            CryptoPP::CBC_Mode<CryptoPP::AES>::Encryption Encryptor((byte*)&pwd[0], pwd.size(), iv);

            StreamTransformationFilter f1(Encryptor, new FileSink(enc));
            ChannelSwitch cs;
            cs.AddDefaultRoute(f1);

            FileSource(filename, true /*pumpAll*/, new Redirector(cs));

        }
        catch(const Exception& ex)
        {
            std::cerr << ex.what() << std::endl;
            return false;
        }
    return true;
}

Remember the length of the passphrase should be of 16 bytes.