Recursive encryption with CipherOutputStream gives an empty byte[]

Question

I'd like to encrypt a string multiple times. But I don't know why I'm ending with an empty byte array. One public key is ok but adding another one returns an empty result.. Does anyone know why ?

private static byte[] encrypt(LinkedList<PublicKey> keys, byte[] input) throws Exception {
    System.out.println("Input length : " + input.length);
    if (keys.isEmpty()) {
        return input;
    }
    PublicKey publicKey = keys.poll();
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    Cipher cipher = Cipher.getInstance("RSA");
    cipher.init(Cipher.ENCRYPT_MODE, publicKey);
    try (CipherOutputStream cipherOutputStream = new CipherOutputStream(byteArrayOutputStream, cipher)) {
        cipherOutputStream.write(input);
    }
    byteArrayOutputStream.flush();
    byteArrayOutputStream.close();
    return encrypt(keys, byteArrayOutputStream.toByteArray());
}

public static void main(String[] args) throws Exception {
    KeyPair pair1 = createPair();
    KeyPair pair2 = createPair();
    LinkedList<PublicKey> keys = new LinkedList<>();
    keys.add(pair1.getPublic());
    keys.add(pair2.getPublic());
    byte[] result = encrypt(keys, "Just testing".getBytes(Charset.forName("UTF-8")));
    System.out.println(new String(result, Charset.forName("UTF-8")));
}

public static KeyPair createPair() throws NoSuchAlgorithmException {
    KeyPairGenerator keyPairGen = KeyPairGenerator.getInstance("RSA");
    keyPairGen.initialize(2048);
    return keyPairGen.generateKeyPair();

}

The output is

Input length : 12
Input length : 256
Input length : 0

After Topaco' answer.. a working version is :

private static BufferedInputStream encryptS(LinkedList<PublicKey> keys, BufferedInputStream inputStream) throws Exception {
    if (keys.isEmpty()) {
        return inputStream;
    }
    PublicKey publicKey = keys.poll();
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
    cipher.init(Cipher.ENCRYPT_MODE, publicKey);
    int currentPos = 0;
    while (inputStream.available() > 0) {
        int sizeToRead = Math.min(inputStream.available(), 245);
        try (CipherOutputStream cipherOutputStream = new CipherOutputStream(byteArrayOutputStream, cipher)) {
            byte[] array = new byte[sizeToRead];
            inputStream.read(array, 0, sizeToRead);
            cipherOutputStream.write(array);
            currentPos += sizeToRead;
        }
    }
    byteArrayOutputStream.close();
    return encryptS(keys, new BufferedInputStream(new ByteArrayInputStream(byteArrayOutputStream.toByteArray())));
}

Closing the byte array output stream after you've already closed the cipher outpiut stream is redundant, as is flush hefore close. — user207421, Sep 21 '19 at 02:22
You **cannot** encrypt arbitrary length data with RSA. Use a [Hybrid Cryptosystem](https://en.m.wikipedia.org/wiki/Hybrid_cryptosystem) to encrypt data of arbitrary length and preserve the semantics of async cryptography. — Boris the Spider, Sep 21 '19 at 08:52
@BoristheSpider No-ops in the case of `ByteArrayOutputStream`, but what I wrote is always true. — user207421, Sep 21 '19 at 09:28

score 5 · Accepted Answer · edited Oct 07 '21 at 08:14

For RSA, the following must be taken into account:

The length of the message plus padding must not exceed the key length (= size of the modulus) [0], [1], [2], [3]. Padding means that additional bytes are added to the message according to a certain scheme [4].
The length of the ciphertext corresponds to the key length (=size of the modulus), [5].

This means that already after the first encryption the maximum allowed length is reached. Thus, without padding the maximum length is not exceeded, with padding it is exceeded.

Creating the cipher instance with

Cipher.getInstance("RSA")

corresponds to

Cipher.getInstance("RSA/ECB/PKCS1Padding")

for the SunJCE-Provider ([6], [7]), i.e. PKCS1 v1.5 padding is used with a padding of at least 11 characters, so that with a key size of 256 bytes the maximum size of the message must not exceed 245 bytes.

This is the reason why the recursive encryption in the current code doesn't work. If the cipher instance is created with

Cipher.getInstance("RSA/ECB/NoPadding")

(no padding used), the current code works.

However, for security reasons a padding must always be used in practice!

Recursive encryption with CipherOutputStream gives an empty byte[]

1 Answers1