Why would web crypto SHA-256 encrypt message while SHA-512 fail sometimes?

Question

I encountered this issue when trying to encrypt some message using RSA-OAEP. To my surprise (as a beginner in crypto) that in some condition SHA-256 works while SHA-512 doesn't.

I couldn't find any helpful info from https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/encrypt#rsa-oaep

Below is some modified demo code from https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey


let message = `{"url":"https://stackoverflow.com/questions/8936984/uint8array-to-string-in-javascript","title":"Uint8Array to string in Javascript - Stack Overflow"}`
let message2 = `just some tree`; // If using SHA-512, this shorter message works but above doesn't

let enc = new TextEncoder()
let dataArray = enc.encode(message)

let keyPair = await window.crypto.subtle.generateKey(
  {
    name: "RSA-OAEP",
    modulusLength: 2048,
    publicExponent: new Uint8Array([1, 0, 1]),
    hash: "SHA-512", // If switched to SHA-256 everything works
  },
  true,
  ["encrypt", "decrypt"]
)
let ciphertext = await window.crypto.subtle.encrypt(
  {
    name: 'RSA-OAEP',
  },
  keyPair.publicKey,
  dataArray
)
let buffer = new Uint8Array(ciphertext, 0, 5);
console.log(`Successful, First 5 bytes: ${buffer}`);

In the above demo code, if the hash function is switched to SHA-256 everything works, but fails to execute if is SHA-512

I have tried a shorter message2. And to my surprised it worked for both SHA-256 and SHA-512

I have tested both async await and .then style. That doesn't seem to be the issue.

I would assume this is related to some kind of length or character limitation? So does that mean I have to use SHA-256? Also curious to learn whether this is a bug or intended feature, if intended what's the official length limitation?

score 2 · Accepted Answer · answered Mar 04 '23 at 11:12

With RSA, only short messages can be encrypted. The maximum message length results from the key size (size of the modulus) minus a value that depends on the padding.
In the case of OAEP, the latter depends on the digest used and is in bytes: 2 * (hLen + 1), where hLen is the output size of the digest in bytes (see here).

So for SHA-256, for a 2048 bits (256 bytes) key, the maximum message length in bytes is: 256 - 2 * (32 + 1) = 190 and for SHA-512: 256 - 2 * (64 + 1) = 126.
In your example, message is UTF-8 encoded 150 bytes, so only SHA-256 can be applied, but not SHA-512. In contrast, message2 has UTF-8 encoded a length of 14 bytes, so both digests can be used, SHA-256 and SHA-512.

Note that longer plaintexts are not encrypted with RSA, but with a hybrid encryption.

Why would web crypto SHA-256 encrypt message while SHA-512 fail sometimes?

1 Answers1