Chilkat iOS10+ RSA compatibility issue

Question

I'm using native RSA in one iOS app and Chilkat RSA library in another. On a native iOS side I encrypt (OAEP SHA256) data with the following function:

    static func encryptWithKey(_ data: Data, rsaKey: SecKey) -> Data? {
        let algorithm = SecKeyAlgorithm.rsaEncryptionOAEPSHA256

        guard SecKeyIsAlgorithmSupported(rsaKey, .encrypt, algorithm) else { return nil }

        var error: Unmanaged<CFError>?
        let encryptedData = SecKeyCreateEncryptedData(rsaKey, algorithm, data as CFData, &error)

        if let encryptionError = error {
            print(encryptionError.takeRetainedValue())
        }

        return encryptedData as Data?
    }

and then in another app, I decrypt this data with Chilkat library:

static func decrypt(base64 encryptedText: String, with xmlKey: String) -> Data? {
    let privateKey = CkoPrivateKey()

    guard let success = privateKey?.loadXml(xmlKey), success else { return nil }
    guard let rsa = CkoRsa() else { return nil }

    rsa.unlockComponent(Chilkat.key)

    rsa.oaepPadding  = true
    rsa.littleEndian = false
    rsa.oaepHash     = "sha256"
    rsa.encodingMode = "base64"

    rsa.importPrivateKeyObj(privateKey)

    if let decryptedData = rsa.decryptBytesENC(encryptedText, bUsePrivateKey: true) {
        return decryptedData
    } else {
        print(rsa.lastErrorText)
    }

    return nil
}

Even I used one private / public keys pair I'm getting an error (from rsa.lastErrorText):

"ChilkatLog:
DecryptBytesENC(7ms):
    DllDate: Feb  1 2018
    ChilkatVersion: 9.5.0.72
    UnlockPrefix: XXXXXXXXXXXX
    Architecture: Little Endian; 64-bit
    Language: IOS C/C++/Swift/Objective-C
    VerboseLogging: 1
    usePrivateKey: 1
    Component successfully unlocked using purchased unlock code.
    rsaDecryptBytes(7ms):
    rsa_decrypt(7ms):
    KeyType: Private
    InputSize: 256
    Padding: OAEP
    OaepHashAlg: SHA-256
    MgfHashAlg: SHA-1
    ParamLen: 0
    ModulusBitLen: 2048
    inlen: 256
    modulus_bytelen: 256
    modulus_bitlen: 2048
    bigEndian: 0
    Byte swapping from big-endian to little-endian
    padding: OAEP
    No leading zero byte for OAEP decoding.
    OAEP decoding failed.
    --rsa_decrypt
    --rsaDecryptBytes
    Failed.
    --DecryptBytesENC
    --ChilkatLog"

Any ideas?

score 1 · Answer 1 · answered May 15 '18 at 12:24

I added some extra logging so we can see the bytes (in hex) after the RSA decrypt but before OAEP unpadding. After RSA decrypt (but before OAEP unpadding), we should see bytes in this format:

0x00  || maskedseed || maskedDB

In other words, the 1st byte of the result SHOULD be a 0 byte. If the RSA private key used to decrypt did not correspond to the public key used to encrypt, then we would see random bytes. (i.e. the 1st byte would not be 0). The maskedseed and maskedDB will still appear as random bytes.

score 0 · Answer 2 · answered May 17 '18 at 02:28

The problem was in OAEP Mgf hash function. If you set oaepMgfHash = "sha256" it becomes compatible with native iOS RSA implementation. In addition, Chilkat library provides an automatic fallback to a proper oaepMgfHash function in case you've specified it incorrectly. Thanks, Matt!

Chilkat iOS10+ RSA compatibility issue

2 Answers2