Java sign/verify keys and Javascript WebCrypto verification fails

Question

I'm trying to

1. generate sign/verification keys (RSA)
2. sign a value (using those keys) on a Java web application (lets call server-side)
3. in order to a web client to verify - public-key imported as RSASSA-PKCS1-v1_5 + SHA-256, (in a browser, using WebCrypto API / client-side)

I'm having problems verifying the signed value (signed in the Java server-side) even though the public sign/verify key is successfully imported as a JWK in the client side.

I was wondering if there is any algorithm compatibility issue in any of the steps (OpenSSL, Java or Javascript) that I may be encountering.

The OpenSSL commands used to generate the keys

openssl genrsa -out privatekey.pem 2048
openssl rsa -in privatekey.pem -pubout > publickey.pub
openssl pkcs8 -topk8 -inform PEM -outform DER -in privatekey.pem -out privatekey-pkcs8.pem

Importing keys with Java (server-side)

public static KeyPair generateSignKeyPair() throws ... {
    byte[] privBytes = b64ToByteArray(PRIVATE_KEY_PEM_VALUE);
    byte[] pubBytes = b64ToByteArray(PUBLIC_KEY_PEM_VALUE);

    // private key
    KeySpec keySpec = new PKCS8EncodedKeySpec(privBytes);
    KeyFactory keyFactory = KeyFactory.getInstance("RSA");
    PrivateKey privateKey = keyFactory.generatePrivate(keySpec);

    // public key (javaPubSignKey)
    X509EncodedKeySpec X509publicKey = new X509EncodedKeySpec(pubBytes);
    PublicKey publicKey = keyFactory.generatePublic(X509publicKey);

    return new KeyPair(publicKey, privateKey);
}

Signing a value with Java (server-side)

 public static byte[] generateSignature(PrivateKey signPrivateKey, byte[] data) throws ... {
    Signature dsa = Signature.getInstance("SHA256withRSA");
    dsa.initSign(signPrivateKey);
    dsa.update(data);
    return dsa.sign();
}

Send them to a web-app for the WebCrypto API to verify as a client/browser (the client is aware of the publicKey generated in the first step).

// Import public sign/verify key (javaPubSignVerifyKey)
var signatureAlgorithm = {
    name: 'RSASSA-PKCS1-v1_5',
    modulusLength: 2048,
    publicExponent: new Uint8Array([0x01, 0x00, 0x01]),
    hash: {
      name: 'SHA-256'
   }
};
// JWK format (1)
crypto.subtle.importKey(
    'jwk', javaPubSignVerifyKey, signatureAlgorithm, false, ['verify']
).then(success, error);

function success(key) {
    signatureVerifyPublicKey = key;
}

Note (1): on the Java side, I'm using com.nimbusds.jose.jwk.JWK to export the publicKey to JWK format.

The sign key is successfully imported by WebCrypto. But when it comes to the verification, it fails (the verification boolean is false).

crypto.subtle.verify(
      signatureAlgorithm,
      signatureVerifyPublicKey,
      signature,               // bytes in Int8Array format (2)
      data                     // bytes in Int8Array format
    ).then(
       function (valid) {
           // valid === false
       }
    )

Note (2): also note that every example I found on WebCrypto used Uint8Array to represent byte arrays, but since Java generates signed byte-arrays I need to use Int8Array so that the signature values are not contaminated (maybe this is an issue aswell).

EDIT: for reference, it turned out to be another unrelated issue - I was converting the expected data from base64 twice in Javascript without noticing it; naturally the verification failed.

Answer 1

Please, check this simple code based on yours to import a RSA public key (spki) and verify a signature. I have generated the keys and signature using similar Java code

var publicKeyB64 = "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDVdZDEs6htb3oxWstz7q+e5IwIRcptMNJiyemuoNyyjtiOy+0tEodjgo7RVoyUcGU3MysEivqvKdswQZ4KfwQCBLAR8DRzp3biAge5utZcKsQoQaC1rCEplfmzEo5ovIlBcMq5x1BxnrnlwEPRmM7MefRa+OeAOQJcstHcrJFO7QIDAQAB";
var dataB64 = "aGVsbG8=";
var signatureB64 = "aEOmUA7YC5gvF6QgH+TMg0erY5pzr83nykZGFtyGOOe+6ld+MC4/Qdb608XiNud+pBpzh0wqd6aajOtJim5XEfCH8vUPsv45aSPtukUIQTX00Oc1frIFDQI6jGJ4Q8dQYIwpqsyE2rkGwTDzt1fTTGiw54pLsJXjtL/D5hUEKL8=";
var signatureAlgorithm = {name: 'RSASSA-PKCS1-v1_5',modulusLength: 2048, publicExponent: new Uint8Array([0x01, 0x00, 0x01]),hash: { name: 'SHA-256'  }};

//convert public key, data and signature to ArrayBuffer. 
var publicKey = str2ab(atob(publicKeyB64)); 
var data = str2ab(atob(dataB64));
var signature = str2ab(atob(signatureB64));            

crypto.subtle.importKey("spki", publicKey, signatureAlgorithm, false,["verify"]).
    then(function(key){
        console.log(key);
        return crypto.subtle.verify( signatureAlgorithm, key, signature, data);                    
}).then( function (valid) {
    console.log("Signature valid: "+valid);
}).catch(function(err) {
    alert("Verification failed " + err );
});

I could not reproduce exactly the issue. Using the str2ab utility function you have linked, the code works perfectly.

//Utility function
function str2ab(str) {
  var arrBuff = new ArrayBuffer(str.length);
  var bytes = new Uint8Array(arrBuff);
  for (var iii = 0; iii < str.length; iii++) {
    bytes[iii] = str.charCodeAt(iii);
  }
  return bytes;
}

I suggest to compare both codes to find the differences