4

I'm using the following code in Python + Pycryptodome (Pycrypto fork) to encrypt a message using RSA PKCS#1 OAEP SHA256 (RSA/ECB/OAEPWithSHA-256AndMGF1Padding): 

from Crypto.Cipher import PKCS1_OAEP
from Cryptodome.Hash import SHA256
cipher = PKCS1_OAEP.new(key=self.key, hashAlgo=SHA256))
ciphertext = cipher.encrypt(cek)

and the following code in Java to decrypt it:

Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher.init(Cipher.DECRYPT_MODE, privateKey);

byte[] cek = cipher.doFinal(ciphertext);

However, I'm getting:

Exception in thread "main" javax.crypto.BadPaddingException: Decryption error
    at sun.security.rsa.RSAPadding.unpadOAEP(RSAPadding.java:499)
    at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:293)
    at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:363)
    at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:389)
    at javax.crypto.Cipher.doFinal(Cipher.java:2165)
Alastair McCormack
  • 26,573
  • 8
  • 77
  • 100

1 Answers1

8

In the Sun JCE, RSA/ECB/OAEPWithSHA-256AndMGF1Padding actually means:

  • Hash = SHA256
  • MGF1 = SHA1

Pycrypto (including Pycryptodome), on the other hand, assumes the following when using PKCS1_OAEP.new(hashAlgo=SHA256):

  • Hash = SHA256
  • MGF1 = SHA256

To make Pycrypto be compatible with Sun JCE, you will need to configure Pycrypto's OAEP MGF1 function to use SHA1 by passing the mgfunc argument:

from Cryptodome.Cipher import PKCS1_OAEP
from Cryptodome.Hash import SHA256, SHA1
from Cryptodome.Signature import pss

cipher = PKCS1_OAEP.new(key=self.key, hashAlgo=SHA256, mgfunc=lambda x,y: pss.MGF1(x,y, SHA1))
ciphertext = cipher.encrypt(cek)

It's worth noting that according to breaking down RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING, BouncyCastle uses SHA256 for both the Hash and MGF1 in the same way Pycrypto does.

Alastair McCormack
  • 26,573
  • 8
  • 77
  • 100
  • I realize this is old but here is how to match it on the java side. Pass this object to the `cipher.init()` function as the parameter spec `new OAEPParameterSpec ( "SHA-256", "MGF1", MGF1ParameterSpec.SHA256, PSource.PSpecified.DEFAULT );` – Zixradoom May 23 '22 at 19:45