Encrypt file in java and decrypt in openssl with key aes

Question

I referred this link to do encrypt a file in java(encrypted file is created successfully)

And later i want to decrypt it using openssl. I tried below commands but none of them seems to work. Please suggest correct command to decrypt in openssl.

openssl aes-256-cbc -d -in sample.crypt -out sample.txt -k testpass

shows "bad magic number"

openssl enc -aes-256-cbc -base64 -pass pass:testpass -d -p -in sample.crypt -out sample.txt

shows "error reading input file"

And here is my encryption java code:

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.Base64;

import javax.crypto.Cipher;
import javax.crypto.CipherOutputStream;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

import static java.nio.charset.StandardCharsets.US_ASCII;
import static java.nio.charset.StandardCharsets.UTF_8;

public class EncryptTest {

    private static final String SALTED_STR = "Salted__";
    private static final byte[] SALTED_MAGIC = SALTED_STR.getBytes(US_ASCII);
    static String password = "testpass";

    public static String encryptfile(Context context, String path, int category) {
            try {
                FileInputStream fis = new FileInputStream(path);
                FileOutputStream fos = new FileOutputStream(path.concat(".crypt"));
                final byte[] pass = password.getBytes(US_ASCII);
                final byte[] salt = (new SecureRandom()).generateSeed(8);

                final byte[] passAndSalt = array_concat(pass, salt);
                byte[] hash = new byte[0];
                byte[] keyAndIv = new byte[0];
                for (int i = 0; i < 3 && keyAndIv.length < 48; i++) {
                    final byte[] hashData = array_concat(hash, passAndSalt);
                    final MessageDigest md = MessageDigest.getInstance("MD5");
                    hash = md.digest(hashData);
                    keyAndIv = array_concat(keyAndIv, hash);
                }

                final byte[] keyValue = Arrays.copyOfRange(keyAndIv, 0, 32);
                final byte[] iv = Arrays.copyOfRange(keyAndIv, 32, 48);
                final SecretKeySpec key = new SecretKeySpec(keyValue, "AES");

                final Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
                cipher.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec(iv));
                CipherOutputStream cos = new CipherOutputStream(fos, cipher);
                int b;
                byte[] d = new byte[8];
                while ((b = fis.read(d)) != -1) {
                    cos.write(d, 0, b);
                }
                cos.flush();
                cos.close();
                fis.close();
                Log.e(TAG, "encrypt done " + path);
            } catch (IOException e) {
                e.printStackTrace();
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
            } catch (NoSuchPaddingException e) {
                e.printStackTrace();
            } catch (InvalidKeyException | InvalidAlgorithmParameterException e) {
                e.printStackTrace();
            }
        }

        return path;
}

score 4 · Accepted Answer · answered Oct 19 '20 at 11:18

OpenSSL expects a special format, namely the ASCII encoded value of Salted__, followed by the 8 bytes salt, followed by the actual ciphertext. In the posted Java code the first block (the ASCII encoded Salted__ plus the salt) is not written, i.e. this must be added, e.g.:

...
final byte[] salt = (new SecureRandom()).generateSeed(8);
fos.write(SALTED_MAGIC);
fos.write(salt);
...

With this change, the ciphertext contains the information and format required for decryption with OpenSSL.

It should be noted that OpenSSL used MD5 as default digest in earlier versions and SHA256 from version v1.1.0 on. For the latter versions, thus a -md md5 must be added to the OpenSSL statement (for earlier versions it's optional).

Both posted OpenSSL statements successfully decrypt a ciphertext created with the (modified) Java code on my machine (in the 2nd statement the -base64 option must be removed, because the Java code does not Base64 encode the ciphertext):

openssl aes-256-cbc -d -in sample.crypt -out sample.txt -k testpass -md md5
openssl enc -aes-256-cbc -pass pass:testpass -d -p -in sample.crypt -out sample.txt -md md5

It should also be mentioned that OpenSSL applies the EVP_BytesToKey() function to derive the key and IV. This function is not a standard and is considered relatively insecure, see e.g. here. New versions of OpenSSL generate a corresponding warning (WARNING : deprecated key derivation used).

Encrypt file in java and decrypt in openssl with key aes

1 Answers1

Linked