RSA Encryption with OAEP between Java and JavaScript

Question

I am trying to encrypt a short string in JavaScript and decrypt it in Java. The decryption fails, and I think it's because of a difference in the block mode and/or padding between the two platforms. I tried encrypting the same string in both Java and JavaScript, and got different results, which indicates that there is indeed a difference. Here is the Java code that creates the key:

KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
kpg.initialize(1024);
KeyPair keyPair = kpg.generateKeyPair();

And here is the Java code that I used to test the encryption:

Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher.init(Cipher.ENCRYPT_MODE, keyPair.getPublic());
byte[] bytes = cipher.doFinal("asdf".getBytes());

I send the public key across to the JavaScript process, and convert it to an ArrayBuffer, with the variable name publicKey. I have verified that the key on the JavaScript side matches the key on the Java side (by exporting it with crypto.subtle.exportKey and examining the bytes). Here is the JavaScript code that I used to test the encryption:

crypto.subtle.importKey('spki', publicKey,
                        {hash: 'SHA-256', name: 'RSA-OAEP'}, true,
                        ['encrypt'])
      .then((key) => {
        crypto.subtle.encrypt({name: 'RSA-OAEP'}, key,
                              new TextEncoder().encode('asdf'))
              .then((buffer) => {

              });
      });

The contents of the byte array in Java and the array buffer in JavaScript are not the same. The settings that I am unsure of are the parameter to Cipher#getInstance on the Java side and the parameters to importKey and encrypt on the JavaScript side. Are there any settings that will work between Java and JavaScript using the built-in classes? Or should I look at third-party libraries (e.g., Bouncy Castle)?

Did you see [this](https://stackoverflow.com/questions/53834903/javas-rsa-ecb-oaepwithsha-256andmgf1padding-equivalent-in-node-js)? — kelalaka, Apr 04 '19 at 22:32
@kelalaka I did not see that question, thank you! Unfortunately, it appears that the only workable suggestion from that question was to use the node-forge package, and I really don't want to use a third-party package if it can be avoided. — Andy King, Apr 05 '19 at 05:22
@AndyKing thanks for the question. I had the same problem, and could find a solution without using third party libraries. Sorry if its too late :). — Chip, Jan 10 '20 at 08:59

score 16 · Answer 1 · answered Jan 10 '20 at 08:57

This is old, but here's an alternate solution if you want to use subtle crypto in javascript, and have control over the java decryption.

Here's how you decrypt in Java assuming you used the original JS code in the question to encrypt:

Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-256"), PSource.PSpecified.DEFAULT);
cipher.init(Cipher.DECRYPT_MODE, privKey, oaepParams);
byte[] decrypted = cipher.doFinal(encryptedBytes)

The issue with the Cipher RSA/ECB/OAEPWithSHA-256AndMGF1Padding is that it uses SHA-1 for the MGF1 Padding by default. Javascript uses SHA-256, which causes the mismatch. By specifying the MGF1ParamterSpec, we can force Java to use the same hashing algorithm as Javascript default.

score 4 · Answer 2 · answered May 16 '20 at 19:08

In addition to @Chip's answer - which was really helpful - I would like to add the following case:

Assume you want to use the following for decryption in Javascript (webcrypto):

window.crypto.subtle.decrypt(
        {
            name: "RSA-OAEP",
            hash: { name: "SHA-512" }
            //label: Uint8Array([...]) //optional
        },
        privateRsaKey, //CryptoKey object containing private RSA key
        encdata //ArrayBuffer containing to be decrypted data
    )
    .catch(function(err){
        console.error(err);
    })

Then you have to use the following OAEPParameterSpec for encryption in Java (and likely vice versa but I did not try that):

OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-512", "MGF1", 
        new MGF1ParameterSpec("SHA-512"), PSource.PSpecified.DEFAULT);

Since @Chip referred to the MGF1 Padding I only, I assumed one would have to use

OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-512", "MGF1", 
        new MGF1ParameterSpec("SHA-256"), PSource.PSpecified.DEFAULT);

but apparently one has to change both hash functions to SHA-512 as shown in my first OAEPParameterSpec code block.

score 0 · Accepted Answer · answered Apr 05 '19 at 20:41

It looks as though the built-in encryption/decryption in JavaScript and Java do not have compatible settings for RSA encryption. A viable solution appears to be the forge library from github (forge on github). The key settings are described on the github page as follows (RSA examples):

// encrypt data with a public key using RSAES-OAEP/SHA-256/MGF1-SHA-1
// compatible with Java's RSA/ECB/OAEPWithSHA-256AndMGF1Padding
var encrypted = publicKey.encrypt(bytes, 'RSA-OAEP', {
  md: forge.md.sha256.create(),
  mgf1: {
    md: forge.md.sha1.create()
  }
});

RSA Encryption with OAEP between Java and JavaScript

3 Answers3

Linked