Java Cipher Decryption on large files

Question

I am trying to decrypt files by Java and below codes work. However it will have OutOfMemory exception when decrypting large files. I tried to change to cipher.update but the program will freeze without any responding.

How can I change this from doFinal to update?

public File decryptDataFile(File inputFile, File outputFile, File keyFile, String correlationId) {
        
        try {
            
            Security.addProvider(new BouncyCastleProvider());
            
            String key = new String(Files.readAllBytes(keyFile.toPath())).trim();
            
            byte[] byteInput = this.getFileInBytes(inputFile);

            Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding", "BC");
            
            byte[] salt = new byte[8];
            System.arraycopy(byteInput, 8, salt, 0, 8);
            SecretKeyFactory fact = SecretKeyFactory.getInstance("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC");
            cipher.init(Cipher.DECRYPT_MODE, fact.generateSecret(new PBEKeySpec(key.toCharArray(), salt, 100)));

            byte[] data = cipher.doFinal(byteInput, 16, byteInput.length-16);
            
            OutputStream os = new FileOutputStream(outputFile);
            os.write(data);
            os.close();
            
            if(outputFile.exists()) {
                return outputFile;
            } else {
                return null;
            }
            
        } catch (IOException | NoSuchAlgorithmException | NoSuchProviderException | NoSuchPaddingException | InvalidKeyException | InvalidKeySpecException | IllegalBlockSizeException | BadPaddingException e) {
            logger.WriteLog(appConfig.getPlatform(), "INFO", alsConfig.getProjectCode(), correlationId, alsConfig.getFunctionId(), "SCAN_DECRYPT", e.getClass().getCanonicalName() + " - " + e.getMessage() );
            return null;
        }

    }

My non-working version:

Security.addProvider(new BouncyCastleProvider());
            
String key = new String(Files.readAllBytes(keyFile.toPath())).trim();
            
byte[] byteInput = this.getFileInBytes(inputFile);

Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding", "BC");
            
byte[] salt = new byte[8];
System.arraycopy(byteInput, 8, salt, 0, 8);
SecretKeyFactory fact = SecretKeyFactory.getInstance("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC");
cipher.init(Cipher.DECRYPT_MODE, fact.generateSecret(new PBEKeySpec(key.toCharArray(), salt, 100)));
            
FileInputStream fis = new FileInputStream(inputFile);
FileOutputStream fos = new FileOutputStream(outputFile);
            
CipherInputStream cis = new CipherInputStream(fis, cipher);
            
int b;
byte[] d = new byte[8];
while((b = cis.read(d)) != -1) {
   fos.write(d, 0, b);
}
fos.flush();
fos.close();
cis.close();
if(outputFile.exists()) {
    return outputFile;
} else {
    return null;
}

Answer 1

Foreword: I couldn't decrypt with your original method a file that was encrypted with your openssl-command

openssl enc -aes-256-cbc -e -salt -in ${tarFile} -out ${encTarFile} -pass file:./${KEY_RANDOM}

but the following method should decode even large files similar to your original method - I tested files up to 1 GB size.

Edit: Regarding the OpenSSL statement, it's worth mentioning that since v1.1.0 the default digest has changed from MD5 to SHA256, so for higher versions the -md MD5 option must be set explicitly for compatibility with the Java code. (thanks to @Topaco).

Please keep in mind that I don't care about correct file paths for

new FileInputStream(inputFile.toPath().toString())
and
new FileOutputStream(outputFile.toPath().toString())

as I'm working locally and with my folder, maybe you have to change the code to "find" your files. As well there is no exception handling in this example.

The code line

byte[] ibuf = new byte[8096];

is defining the buffer that is used - a larger buffer makes the decryption faster but consumes more memory (8096 means 8096 byte compared to 1 Gbyte when reading the complete file into memory and causing the out of memory error).

public static File decryptDataFileBuffered(File inputFile, File outputFile, File keyFile, String correlationId) throws IOException, NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException, InvalidKeySpecException, BadPaddingException, IllegalBlockSizeException, InvalidKeyException {
        Security.addProvider(new BouncyCastleProvider());
        String key = new String(Files.readAllBytes(keyFile.toPath())).trim();
        byte[] salt = new byte[8];
        byte[] salted = new byte[8]; // text SALTED__
        try (FileInputStream in = new FileInputStream(inputFile.toPath().toString()); // i don't care about the path as all is lokal
             FileOutputStream out = new FileOutputStream(outputFile.toPath().toString())) // i don't care about the path as all is lokal
        {
            byte[] ibuf = new byte[8096]; // thats the buffer used - larger is faster
            int len;
            in.read(salted);
            in.read(salt);
            SecretKeyFactory fact = SecretKeyFactory.getInstance("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC");
            SecretKey secretKey = fact.generateSecret(new PBEKeySpec(key.toCharArray(), salt, 100));
            System.out.println("secretKey length: " + secretKey.getEncoded().length + " data: " + bytesToHex(secretKey.getEncoded()));
            Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
            cipher.init(Cipher.DECRYPT_MODE, secretKey);
            while ((len = in.read(ibuf)) != -1) {
                byte[] obuf = cipher.update(ibuf, 0, len);
                if (obuf != null)
                    out.write(obuf);
            }
            byte[] obuf = cipher.doFinal();
            if (obuf != null)
                out.write(obuf);
         } catch (IOException | BadPaddingException | IllegalBlockSizeException e) {
            e.printStackTrace();
         }
        if (outputFile.exists()) {
            return outputFile;
        } else {
            return null;
        }
    }

Edit2: As commented by @Topaco the usage of CipherInput/OutputStream shortens the code and makes it better readable, so here is the code:

public static File decryptDataFileBufferedCipherInputStream (File inputFile, File outputFile, File keyFile, String correlationId) throws
            IOException, NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException, InvalidKeySpecException, InvalidKeyException
    {
        Security.addProvider(new BouncyCastleProvider());
        String key = new String(Files.readAllBytes(keyFile.toPath())).trim();
        byte[] salt = new byte[8];
        byte[] salted = new byte[8]; // text SALTED__
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        try (FileInputStream in = new FileInputStream(inputFile.toPath().toString()); // i don't care about the path as all is lokal
             CipherInputStream cipherInputStream = new CipherInputStream(in, cipher);
             FileOutputStream out = new FileOutputStream(outputFile.toPath().toString())) // i don't care about the path as all is lokal
        {
            byte[] buffer = new byte[8192];
            in.read(salted);
            in.read(salt);
            SecretKeyFactory fact = SecretKeyFactory.getInstance("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC");
            SecretKey secretKey = fact.generateSecret(new PBEKeySpec(key.toCharArray(), salt, 100));
            System.out.println("secretKey length: " + secretKey.getEncoded().length + " data: " + bytesToHex(secretKey.getEncoded()));
            cipher.init(Cipher.DECRYPT_MODE, secretKey);
            int nread;
            while ((nread = cipherInputStream.read(buffer)) > 0) {
                out.write(buffer, 0, nread);
            }
            out.flush();
        }
        if (outputFile.exists()) {
            return outputFile;
        } else {
            return null;
        }
    }