Encrypt in Java using RSA/ECB/OAEPWithSHA-256AndMGF1Padding, decrypt in c# using OAEPSHA256 Padding

Question

I have a situation where a Java program encrypts text using RSA/ECB/OAEPWithSHA-256AndMGF1Padding.

I need to decrypt it in c#.

Encryption and decryption work fine in Java.

Encryption in Java and decryption in c# with RSA/ECB/OAEPWithSHA-1AndMGF1Padding works absolutely fine.

However, with RSA/ECB/OAEPWithSHA-256AndMGF1Padding encryption in Java and decryption with OaepSHA256 in C# gives me the error : The parameter is incorrect.

Java Code for encrypt:

public static String encrypt(KeyPair keypair, String data) throws NoSuchAlgorithmException, NoSuchPaddingException,
            InvalidKeyException, IllegalBlockSizeException, BadPaddingException, UnsupportedEncodingException {
        Cipher c = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
        //Cipher c = Cipher.getInstance("RSA/ECB/OAEPWithSHA-1AndMGF1Padding");
        c.init(Cipher.ENCRYPT_MODE, keypair.getPublic());
        return Base64.getEncoder().encodeToString(c.doFinal(data.getBytes()));
    }

C # code for decrypt :

public string DecryptRsa(byte[] encryptedBytes, X509Certificate2 x509Certificate2, RSAEncryptionPadding rSAEncryptionPadding)
        {
            var text = string.Empty;

            using (RSACng csp = (RSACng)x509Certificate2.GetRSAPrivateKey())
            {

                byte[] bytesDecrypted = csp.Decrypt(encryptedBytes, rSAEncryptionPadding);
                text = Encoding.UTF8.GetString(bytesDecrypted);
            }
            return text;
        }

What am i doing wrong? Please help.

Topaco · Accepted Answer · 2021-07-05T20:43:33.057

OAEP uses two digests, the OAEP digest and the MGF1 digest, see RFC8017.

The SunJCE provider specifies with RSA/ECB/OAEPWithSHA-256AndMGF1Padding the OAEP digest as SHA256, while the MGF1 digest defaults to SHA1, see here. The C# code, on the other hand, specifies with OaepSHA256 both digests as SHA256. Therefore, both codes are incompatible.

The fix is to either explicitly specify the digests in Java with OAEPParameterSpec (which should always be done anyway for this very reason). On the C# side, no fix is possible with on board means, since a separate specification of both digests is not supported. But BouncyCastle can be used which supports this.

Fix, Java code side (SHA256 for both digests):

RSAPublicKey publicKey = ...
OAEPParameterSpec oaepParameterSpec = new OAEPParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, PSource.PSpecified.DEFAULT);
Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
cipher.init(Cipher.ENCRYPT_MODE, publicKey, oaepParameterSpec);
byte[] ciphertext = cipher.doFinal(plaintext);

Fix, C# code side, using BouncyCastle (SHA256 for OAEP digest, SHA1 for MGF1 digest):

using Org.BouncyCastle.Crypto.Digests;
using Org.BouncyCastle.Crypto.Engines;
using Org.BouncyCastle.Crypto.Encodings;
using Org.BouncyCastle.Crypto.Parameters;
...
RsaKeyParameters privateKey = ...
OaepEncoding oaepEncoding = new OaepEncoding(new RsaEngine(), new Sha256Digest(), new Sha1Digest(), null);
oaepEncoding.Init(false, privateKey);
byte[] decrypted = oaepEncoding.ProcessBlock(ciphertext, 0, ciphertext.Length);

Thanks very much. Unfortunately, I do not have any control over the Java code as it's third party code calling my API using my public key for encryption. I tried using BouncyCastle. Exactly the same code as suggested by you above. It gives me a very cryptic "data wrong" error. — MethodToChaos, Jul 06 '21 at 01:48
I am sorry. When I said I did use BouncyCastle I used Sha256Digest for both digests to initialize OaepEncoding. Will try with the other hash as you suggested. Thanks very much. — MethodToChaos, Jul 06 '21 at 01:58
This works with my Java/C# Encryption/Decryption (Which was not working earlier.). So i willl mark this as answer. However the third party encrypted string still throws "data wrong" error in BouncyCastle C#. Must be something wrong with what they are doing @ encryption. — MethodToChaos, Jul 06 '21 at 03:55
Great answer. I had to implement a .NET version of `RSA/ECB/OAEPWITHSHA-512ANDMGF1PADDING` which ended up being `new OaepEncoding(new RsaEngine(), new Sha512Digest(), new Sha1Digest(), null);`. — rgvlee, Jul 19 '23 at 08:44

Encrypt in Java using RSA/ECB/OAEPWithSHA-256AndMGF1Padding, decrypt in c# using OAEPSHA256 Padding

1 Answers1