OpenSSL decrypt fails but error code is 0

Question

When I try to decrypt an encrypted S/MIME message using CMS in OpenSSL, the decrypt method returns me 0 which stands for didn't succeed.

OpenSSL.org says..

CMS_decrypt() returns either 1 for success or 0 for failure. The error can be obtained from ERR_get_error(3)

When I run this...

out = BIO_new(BIO_s_mem());
if (!out)
        assert(false);

int error = CMS_decrypt(cms, rkey, rcert, out, NULL, 0);
    if (!error) {
    fprintf(stderr, "Error Decrypting Data\n");
    printf("error code: %d\n", ERR_get_error());
    ERR_print_errors_fp(stderr);
    assert(false);
}

... the error variable is 0 which means an error occurred and the error code from ERR_get_error() is also 0. Additionally ERR_print_errors_fp() doesn't print anything which means there was no error.

The output from the aforementioned code:

Error Decrypting Data
error code: 0
Assertion failed: (false)

Does anyone have a suggestion what's going wrong here? Thanks

does it decrypt successfully? If so it may be worth checking that an error returns 0 as intended (return values may have been accidentally swapped) or just use the "ERR_get_error()" value in the control statement — MD-Tech, Feb 24 '12 at 15:31
@MD-Tech: Nope. :-/ The BIO is empty. Even if I create a file BIO instead of a mem BIO it creates the new file but doesn't write anything in it... — Chris, Feb 24 '12 at 22:29

score 1 · Accepted Answer · answered May 17 '12 at 17:32

After many weeks of trying different approaches and frustration I ended up using PKCS#7 decrypt of OpenSSL. Due CMS is basically based on PKCS#7 it works fine with encryption using CMS and decryption using PKCS7.

The method I have used is pkcs7_decrypt().

OpenSSL decrypt fails but error code is 0

1 Answers1