RSA decryption fine with encoded byte array but fails with encoded String [IllegalBlockSizeException]

Question

we are trying RSA Encryption & Decryption and the issue happens while decrypting. This is our decryption code

Cipher oaepFromInit = Cipher.getInstance("RSA/ECB/OAEPPadding");
OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-1", "MGF1", new MGF1ParameterSpec("SHA-1"), PSpecified.DEFAULT);
oaepFromInit.init(Cipher.DECRYPT_MODE, rsaPrivateKey, oaepParams);
byte[] dec = oaepFromInit.doFinal(encrytpedData.getBytes());

The encrytpedData is like this

s11Pyj5rrVOfOiWtxpGq+K5D+pYi16CyyX/EwKfMErBkHJ4aVlTmnhrfeCS7LEeXgTs3gkFp96I/oTedG/rXxF2hTAmMH40k0joKJbRtzO858/0dcaaE1uNzr/rI0Jj3ebXPLGhefCMNNpyFH5V4ukVo6vtev5Z9U8oNkUQolbX/r5jJJomkKCCnzGoHMdQg5dafj9Sw/qakO13501YBrkxS0i9ca0GZ8Ll42NwkOZuInh+MAu+gYW4vAr284eJsqgLgTp0+MS1tmfwR6EXgspk1nYR/U84P3MBZAdpmD3nxsVV3iVOCUeoqVyd4kw7M2pvXev6hMbMN4P1nnomo8g==

An error exception was thrown saying javax.crypto.IllegalBlockSizeException: Data must not be longer than 256 bytes

We looked into this link getting a IllegalBlockSizeException: Data must not be longer than 256 bytes when using rsa, but the issue was with the way we were feeding the encrytpedData.

This is a code we got online

KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
kpg.initialize(2048);
KeyPair kp = kpg.generateKeyPair();
RSAPublicKey pubkey = (RSAPublicKey) kp.getPublic();
RSAPrivateKey privkey = (RSAPrivateKey) kp.getPrivate();
        
// --- encrypt given algorithm string
Cipher oaepFromAlgo = Cipher.getInstance("RSA/ECB/OAEPWithSHA-1AndMGF1Padding");
oaepFromAlgo.init(Cipher.ENCRYPT_MODE, pubkey);
byte[] ct = oaepFromAlgo.doFinal("chakka".getBytes(StandardCharsets.UTF_8));

// --- decrypt given OAEPParameterSpec
Cipher oaepFromInit = Cipher.getInstance("RSA/ECB/OAEPPadding");
OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-1", "MGF1", new MGF1ParameterSpec("SHA-1"), PSpecified.DEFAULT);
oaepFromInit.init(Cipher.DECRYPT_MODE, privkey, oaepParams);
byte[] pt = oaepFromInit.doFinal(ct);
System.out.println("Printing decoded string");
System.out.println(new String(pt, StandardCharsets.UTF_8));

Here they are supplying the encrypted array directly and works fine, but in our case we are getting encrytpedData as String. Both the codes are same but the supply of encrypted data differs.

byte[] dec = oaepFromInit.doFinal(encrytpedData.getBytes());

This is where we are getting the exception, so we tried converting the encrytpedData to byte[] in a different way via these two links

1. RSA encyrption - converting between bytes array and String [duplicate]
2. IllegalBlockSizeException when trying to encrypt and decrypt a string with AES

So we tried like this

1. byte[] dec = oaepFromInit.doFinal(Base64.getDecoder().decode(encrytpedData)); which resulted in this javax.crypto.BadPaddingException: Message is larger than modulus.
2. byte[] dec = oaepFromInit.doFinal(encrytpedData.getBytes(StandardCharsets.UTF_8)); which gave the same old exception javax.crypto.IllegalBlockSizeException: Data must not be longer than 256 bytes.

Any help would be appreciated

Answer 1

The below full example code is taken from my private cross-platform project and should work. It encrypts a string using RSA encryption with a 2048 key pair and uses OAEP padding with SHA-1 as hash.

The encrypted data are in base64 encoding. For demonstration purpose I'm using static (hardcoded) keys in PEM-format - never ever do this in production.

You can run the code online here: https://repl.it/@javacrypto/CpcJavaRsaEncryptionOaepSha1String

this is the output:

RSA 2048 encryption OAEP SHA-1 string
plaintext: The quick brown fox jumps over the lazy dog

* * * encrypt the plaintext with the RSA public key * * *
ciphertextBase64: pSPayOK79NMPLpaQOp8L7dOTdLeS+jVU5cKPF5mDOvuKtnOjf/NYfgsf1JWa5Ud/fzsXrSN8I/8KAs/freagOYflv6PGUHZ7cYk1iX6sO5cmdD0Mfglj39NxczbqCK3FG20hQfa01ZaOu/dV8+QvVx851ph1nEl8arNML5ohhjIdZTxR3olGouGzDJmuJkDlv2fJICP7sdMnvl7t21QI701I2xR01eatD5src5fY//EIOEjSoTesCBZwEUz1S3UpclNqGONAqaYqlscdTyAlY3Hg8smJRqKWk7ZWKSTYsocBbFeTvcNy75LG/xyILM0l4U+NnwGUypkjR2vJRGVQtw==

* * * decrypt the ciphertext with the RSA private key * * *
ciphertextReceivedBase64: pSPayOK79NMPLpaQOp8L7dOTdLeS+jVU5cKPF5mDOvuKtnOjf/NYfgsf1JWa5Ud/fzsXrSN8I/8KAs/freagOYflv6PGUHZ7cYk1iX6sO5cmdD0Mfglj39NxczbqCK3FG20hQfa01ZaOu/dV8+QvVx851ph1nEl8arNML5ohhjIdZTxR3olGouGzDJmuJkDlv2fJICP7sdMnvl7t21QI701I2xR01eatD5src5fY//EIOEjSoTesCBZwEUz1S3UpclNqGONAqaYqlscdTyAlY3Hg8smJRqKWk7ZWKSTYsocBbFeTvcNy75LG/xyILM0l4U+NnwGUypkjR2vJRGVQtw==
decryptedtext: The quick brown fox jumps over the lazy dog

Security warning: the code does not have any exception handling, uses static (hardcoded) keys and is for educational purpose only.

code:

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.OAEPParameterSpec;
import javax.crypto.spec.PSource;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.*;
import java.security.spec.MGF1ParameterSpec;
import java.security.spec.PKCS8EncodedKeySpec;
import java.security.spec.X509EncodedKeySpec;
import java.util.Base64;

public class Main {
    public static void main(String[] args) throws GeneralSecurityException, IOException {
        System.out.println("RSA 2048 encryption OAEP SHA-1 string");

        String dataToEncryptString = "The quick brown fox jumps over the lazy dog";
        byte[] dataToEncrypt = dataToEncryptString.getBytes(StandardCharsets.UTF_8);
        System.out.println("plaintext: " + dataToEncryptString);

        // # # # usually we would load the private and public key from a file or keystore # # #
        // # # # here we use hardcoded keys for demonstration - don't do this in real programs # # #
        String filenamePrivateKeyPem = "privatekey2048.pem";
        String filenamePublicKeyPem = "publickey2048.pem";

        // encryption
        System.out.println("\n* * * encrypt the plaintext with the RSA public key * * *");
        PublicKey publicKeyLoad = getPublicKeyFromString(loadRsaPublicKeyPem());
        // use this in production
        //PublicKey publicKeyLoad = getPublicKeyFromString(loadRsaKeyPemFile(filenamePublicKeyPem));
        String ciphertextBase64 = base64Encoding(rsaEncryptionOaepSha1(publicKeyLoad, dataToEncrypt));
        System.out.println("ciphertextBase64: " + ciphertextBase64);

        // transport the encrypted data to recipient

        // receiving the encrypted data, decryption
        System.out.println("\n* * * decrypt the ciphertext with the RSA private key * * *");
        String ciphertextReceivedBase64 = ciphertextBase64;
        //String ciphertextReceivedBase64 = "l4G3O42LtjI9KkzvcF7SQcpqrkOMJw1sWVuI3FCZ1g+Sp/t05E3ZEXyVV/FnadkKVgmpWBifaYPEdKNBTbuts2F1DfrDz1v14lKlMOcqkJB8OmJUAiKJ1ic414R7M5fECKruqkzOdKlTtdb3MI49Ygrzd/cJxOGEvONo3DAOq1kZvZdmyW+K8m807g2qoy833EHyj9NjVZHuDzXi8fMxbIAI5MrN8ykXZBxkFOAGiITEFbGPxu6gBOPJKsPWJ0SVU53CiI1YwGc76/ov4c7FIA1ZeVsJXKo8CEfYuUc7PKIJ3e5Z7CbiNgr4Z5720Xbi0drUBk/LlYq6m8s/zEIMaQ==";
        System.out.println("ciphertextReceivedBase64: " + ciphertextReceivedBase64);
        PrivateKey privateKeyLoad = getPrivateKeyFromString(loadRsaPrivateKeyPem());
        // use this in production
        //PrivateKey privateKeyLoad = getPrivateKeyFromString(loadRsaKeyPemFile(filenamePrivateKeyPem));
        byte[] ciphertextReceived = base64Decoding(ciphertextReceivedBase64);
        byte[] decryptedtextByte = rsaDecryptionOaepSha1(privateKeyLoad, ciphertextReceived);
        System.out.println("decryptedtext: " + new String(decryptedtextByte, StandardCharsets.UTF_8));
    }

    public static byte[] rsaEncryptionOaepSha1 (PublicKey publicKey, byte[] plaintextByte) throws NoSuchAlgorithmException,
            InvalidKeyException, IllegalBlockSizeException, BadPaddingException, NoSuchPaddingException, InvalidAlgorithmParameterException {
        byte[] ciphertextByte = null;
        Cipher encryptCipher = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-1ANDMGF1PADDING");
        //OAEPParameterSpec oaepParameterSpecJCE = new OAEPParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, PSource.PSpecified.DEFAULT);
        // note: SHA1 is the default for Java
        OAEPParameterSpec oaepParameterSpecJCE = new OAEPParameterSpec("SHA1", "MGF1", MGF1ParameterSpec.SHA1, PSource.PSpecified.DEFAULT);
        encryptCipher.init(Cipher.ENCRYPT_MODE, publicKey, oaepParameterSpecJCE);
        ciphertextByte = encryptCipher.doFinal(plaintextByte);
        return ciphertextByte;
    }

    public static byte[] rsaDecryptionOaepSha1 (PrivateKey privateKey, byte[] ciphertextByte) throws NoSuchAlgorithmException,
            NoSuchPaddingException, InvalidKeyException, BadPaddingException, IllegalBlockSizeException, InvalidAlgorithmParameterException {
        byte[] decryptedtextByte = null;
        Cipher decryptCipher = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-1ANDMGF1PADDING");
        //OAEPParameterSpec oaepParameterSpecJCE = new OAEPParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, PSource.PSpecified.DEFAULT);
        OAEPParameterSpec oaepParameterSpecJCE = new OAEPParameterSpec("SHA1", "MGF1", MGF1ParameterSpec.SHA1, PSource.PSpecified.DEFAULT);
        decryptCipher.init(Cipher.DECRYPT_MODE, privateKey, oaepParameterSpecJCE);
        decryptedtextByte = decryptCipher.doFinal(ciphertextByte);
        return decryptedtextByte;
    }

    private static String base64Encoding(byte[] input) {
        return Base64.getEncoder().encodeToString(input);
    }
    private static byte[] base64Decoding(String input) {
        return Base64.getDecoder().decode(input);
    }

    private static String loadRsaPrivateKeyPem() {
        // this is a sample key - don't worry !
        return "-----BEGIN PRIVATE KEY-----\n" +
                "MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDwSZYlRn86zPi9\n" +
                "e1RTZL7QzgE/36zjbeCMyOhf6o/WIKeVxFwVbG2FAY3YJZIxnBH+9j1XS6f+ewjG\n" +
                "FlJY4f2IrOpS1kPiO3fmOo5N4nc8JKvjwmKtUM0t63uFFPfs69+7mKJ4w3tk2mSN\n" +
                "4gb8J9P9BCXtH6Q78SdOYvdCMspA1X8eERsdLb/jjHs8+gepKqQ6+XwZbSq0vf2B\n" +
                "MtaAB7zTX/Dk+ZxDfwIobShPaB0mYmojE2YAQeRq1gYdwwO1dEGk6E5J2toWPpKY\n" +
                "/IcSYsGKyFqrsmbw0880r1BwRDer4RFrkzp4zvY+kX3eDanlyMqDLPN+ghXT1lv8\n" +
                "snZpbaBDAgMBAAECggEBAIVxmHzjBc11/73bPB2EGaSEg5UhdzZm0wncmZCLB453\n" +
                "XBqEjk8nhDsVfdzIIMSEVEowHijYz1c4pMq9osXR26eHwCp47AI73H5zjowadPVl\n" +
                "uEAot/xgn1IdMN/boURmSj44qiI/DcwYrTdOi2qGA+jD4PwrUl4nsxiJRZ/x7PjL\n" +
                "hMzRbvDxQ4/Q4ThYXwoEGiIBBK/iB3Z5eR7lFa8E5yAaxM2QP9PENBr/OqkGXLWV\n" +
                "qA/YTxs3gAvkUjMhlScOi7PMwRX9HsrAeLKbLuC1KJv1p2THUtZbOHqrAF/uwHaj\n" +
                "ygUblFaa/BTckTN7PKSVIhp7OihbD04bSRrh+nOilcECgYEA/8atV5DmNxFrxF1P\n" +
                "ODDjdJPNb9pzNrDF03TiFBZWS4Q+2JazyLGjZzhg5Vv9RJ7VcIjPAbMy2Cy5BUff\n" +
                "EFE+8ryKVWfdpPxpPYOwHCJSw4Bqqdj0Pmp/xw928ebrnUoCzdkUqYYpRWx0T7YV\n" +
                "RoA9RiBfQiVHhuJBSDPYJPoP34kCgYEA8H9wLE5L8raUn4NYYRuUVMa+1k4Q1N3X\n" +
                "Bixm5cccc/Ja4LVvrnWqmFOmfFgpVd8BcTGaPSsqfA4j/oEQp7tmjZqggVFqiM2m\n" +
                "J2YEv18cY/5kiDUVYR7VWSkpqVOkgiX3lK3UkIngnVMGGFnoIBlfBFF9uo02rZpC\n" +
                "5o5zebaDImsCgYAE9d5wv0+nq7/STBj4NwKCRUeLrsnjOqRriG3GA/TifAsX+jw8\n" +
                "XS2VF+PRLuqHhSkQiKazGr2Wsa9Y6d7qmxjEbmGkbGJBC+AioEYvFX9TaU8oQhvi\n" +
                "hgA6ZRNid58EKuZJBbe/3ek4/nR3A0oAVwZZMNGIH972P7cSZmb/uJXMOQKBgQCs\n" +
                "FaQAL+4sN/TUxrkAkylqF+QJmEZ26l2nrzHZjMWROYNJcsn8/XkaEhD4vGSnazCu\n" +
                "/B0vU6nMppmezF9Mhc112YSrw8QFK5GOc3NGNBoueqMYy1MG8Xcbm1aSMKVv8xba\n" +
                "rh+BZQbxy6x61CpCfaT9hAoA6HaNdeoU6y05lBz1DQKBgAbYiIk56QZHeoZKiZxy\n" +
                "4eicQS0sVKKRb24ZUd+04cNSTfeIuuXZrYJ48Jbr0fzjIM3EfHvLgh9rAZ+aHe/L\n" +
                "84Ig17KiExe+qyYHjut/SC0wODDtzM/jtrpqyYa5JoEpPIaUSgPuTH/WhO3cDsx6\n" +
                "3PIW4/CddNs8mCSBOqTnoaxh\n" +
                "-----END PRIVATE KEY-----";
    }

    private static String loadRsaPublicKeyPem() {
        // this is a sample key - don't worry !
        return "-----BEGIN PUBLIC KEY-----\n" +
                "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8EmWJUZ/Osz4vXtUU2S+\n" +
                "0M4BP9+s423gjMjoX+qP1iCnlcRcFWxthQGN2CWSMZwR/vY9V0un/nsIxhZSWOH9\n" +
                "iKzqUtZD4jt35jqOTeJ3PCSr48JirVDNLet7hRT37Ovfu5iieMN7ZNpkjeIG/CfT\n" +
                "/QQl7R+kO/EnTmL3QjLKQNV/HhEbHS2/44x7PPoHqSqkOvl8GW0qtL39gTLWgAe8\n" +
                "01/w5PmcQ38CKG0oT2gdJmJqIxNmAEHkatYGHcMDtXRBpOhOSdraFj6SmPyHEmLB\n" +
                "ishaq7Jm8NPPNK9QcEQ3q+ERa5M6eM72PpF93g2p5cjKgyzzfoIV09Zb/LJ2aW2g\n" +
                "QwIDAQAB\n" +
                "-----END PUBLIC KEY-----";
    }

    public static PrivateKey getPrivateKeyFromString(String key) throws GeneralSecurityException {
        String privateKeyPEM = key;
        privateKeyPEM = privateKeyPEM.replace("-----BEGIN PRIVATE KEY-----", "");
        privateKeyPEM = privateKeyPEM.replace("-----END PRIVATE KEY-----", "");
        privateKeyPEM = privateKeyPEM.replaceAll("[\\r\\n]+", "");
        byte[] encoded = Base64.getDecoder().decode(privateKeyPEM);
        KeyFactory kf = KeyFactory.getInstance("RSA");
        PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encoded);
        PrivateKey privKey = (PrivateKey) kf.generatePrivate(keySpec);
        return privKey;
    }

    public static PublicKey getPublicKeyFromString(String key) throws GeneralSecurityException {
        String publicKeyPEM = key;
        publicKeyPEM = publicKeyPEM.replace("-----BEGIN PUBLIC KEY-----", "");
        publicKeyPEM = publicKeyPEM.replace("-----END PUBLIC KEY-----", "");
        publicKeyPEM = publicKeyPEM.replaceAll("[\\r\\n]+", "");
        byte[] encoded = Base64.getDecoder().decode(publicKeyPEM);
        KeyFactory kf = KeyFactory.getInstance("RSA");
        PublicKey pubKey = (PublicKey) kf.generatePublic(new X509EncodedKeySpec(encoded));
        return pubKey;
    }

    private static String loadRsaKeyPemFile(String filename) throws IOException {
        return new String(Files.readAllBytes(Paths.get(filename)), StandardCharsets.UTF_8);
    }
}