0

Updated based on comments.

I have read several articles on here about similar problems of people trying to solve issues when trying to use a DLL function inside VBA through a declare.

I tried so many variants (byref/byval, long, longptr, string ...) but either excel crashes, either the call doesn't do anything apparently

This is the info I have on the DLL:

'/// <summary> Encrypt the message. </summary>
'/// <param name="penc_key_bin"> The binary encryption key. </param>
'/// <param name="penc_iv_bin"> The binary encryption iv. </param>
'/// <param name="punenc_data"> The unencrypted data buffer. </param>
'/// <param name="unenc_offset"> The offset of the first byte of the unencrypted buffer. </param>
'/// <param name="unenc_len"> Length of the unencrypted data to encrypt. </param>
'/// <param name="penc_data"> [out] The encrypted data buffer. </param>
'/// <param name="enc_offset"> The offset of the first byte of the encrypted data buffer. </param>
'/// <param name="penc_len"> [out] Length of the encrypted data. </param>
'/// <returns> The standard SPC return code. </returns>
'unsigned char aes256_cbc_encrypt_message(
'  unsigned char *penc_key_bin, 
'  unsigned char *penc_iv_bin, 
'  unsigned char *punenc_data, 
'  long unenc_offset, 
'  long unenc_len, 
'  unsigned char *penc_data, 
'  long enc_offset, 
'  long *penc_len)

And this is what I currently have what I feel should best fit .. but causes a crash

Private Declare PtrSafe Function aes256_cbc_encrypt_message Lib "Flexc Encryption.dll" _
    (ByVal penc_key_bin As LongPtr, _
    ByVal penc_iv_bin As LongPtr, _
    ByVal punenc_data As LongPtr, _
    ByVal unenc_offset As Long, _
    ByVal unenc_len As Long, _
    ByVal penc_data As LongPtr, _
    ByVal enc_offset As Long, _
    ByVal penc_len As Long) _
As Long

I use is like here:

Sub test()
    Dim Key() As Byte
    Dim IV() As Byte
    Dim res As Variant
    Dim strDec() As Byte
    Dim strEnc() As Byte
    Dim decOff As Long
    Dim decLen As Long
    
    Dim plainASCII As String
    
    plainASCII = "TESTTESTTESTTESTTESTTESTTESTTEST"
    
    decOff = 0
    decLen = Len(plainASCII)
    
    res = check_dll_activated()
    strDec = stringToByteArr(plainASCII)
    ReDim strEnc(decLen - 1)
    
    
    Key = stringToByteArr(DecodeHex("0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF"))
    IV = stringToByteArr(DecodeHex("00000000000000000000000000000000"))
    res = aes256_cbc_encrypt_message((Key(0)), (IV(0)), (strDec(0)), 0, 32, (strEnc(0)), decOff, decLen)
    'variant 1: res = aes256_cbc_encrypt_message(VarPtr(Key(0)), VarPtr(IV(0)), VarPtr(strDec(0)), 0, 32, VarPtr(strEnc(0)), decOff, decLen)
    'variant 2: res = aes256_cbc_encrypt_message(VarPtr(Key), VarPtr(IV), VarPtr(strDec), 0, 32, VarPtr(strEnc), decOff, decLen)
End Sub

Private Function stringToByteArr(inpStr As String) As Byte()
    Dim outByte() As Byte
    ReDim outByte(Len(inpStr) - 1)
    For i = 1 To Len(inpStr)
        outByte(i - 1) = Asc(Mid(inpStr, i, 1))
    Next i
    'outByte = inpStr
    stringToByteArr = outByte
End Function
Private Function DecodeHex(ByVal HexString As String) As String
    Dim thisChar As String
    Dim ascii As Integer
    Dim ret As String
    'Keep going until we exhaust all the source string
    Do While Len(HexString) > 0
        'Get the next two-digit hex number
        thisChar = Left(HexString, 2)
        'Remove this hex number from the source string
        HexString = Mid(HexString, 3)
        'Get the value in decimal of this hex number
        ascii = CInt(Val("&H" & thisChar))
        'Convert it to a character
        ret = ret & Chr(ascii)
    Loop
    'All done
    DecodeHex = ret
End Function

Any advice is helpful. C.

Christof De Backere
  • 23
  • 6
  • Two things. We don't know what DecodeHex returns. You imply its a string but witthout the code for DecodeHex, we don't know. Secondly, are you sure that the strings provided to strptr are giving you the correct result. Typically unsigned chars are obtained from arrays of Byte having used the appropriate StrConv function to get a unicode or ansi representation of the string. If you are providing unicode when your declare is expecting ansi things won'tt work too well. – freeflow Nov 08 '22 at 15:26
  • I added the DecodeHEx code to be complete. It does return a string. As for the second part of your comment: no idea. I don't even understand the question to be honnest, but that tells more about me than about you. I don't care what representation is used or needed. All I know is the C-like declaration 'unsigned char *', and the byte/char strings that I provide and need back. So a string where 1 char is 1 byte is what I neeed in my application. – Christof De Backere Nov 08 '22 at 15:32
  • Ok.. Then I don't care that you were given the solution but are choosing to ignore it. In the meantime would this be helpful. https://stackoverflow.com/questions/63730549/vba-aes-cbc-encryption – freeflow Nov 08 '22 at 16:36
  • When I read my own 'don't care' after a few minutes it sounded wrong, but the site didn't let me correct it because 'you can only edit a comment for 5 minutes'. So please forgive me the wording (engish is also only my 3rd language, after dutch and french) as I ready did not mean to be 'rude' or 'impolite'. But I'll check your sugeestion with attention. – Christof De Backere Nov 08 '22 at 20:16
  • Thanks for the sugested AES implementation. But I really need to use this dll version, because we must build an API with it, and from what we can see, the AES256 CBC encryption they use in the dll is not 100% a pure AES256 CBC. It seem they do some prefixing/wrapping of the plain text input. So I need to check the dll encruption/decryption with the documented AES256 CBC test vektors. So the dll use is a must in my case – Christof De Backere Nov 08 '22 at 20:17
  • I read your first comment several more times. And I am realy ignorent on those details. I know only that we get a serial stream of bytes as input that we store as 1 lua variable (lua has no type). (yes, lua is where we do it, but need need the dll for testing) That this 'string' is to be decoded in a 'string' of equal length. And that this decoded 'string' is then intepreted as a 'real text string' by converting each byte to the corresponding 'ASCII' character. But I realy have no clue if this is unicode, ansi, byte-array, ... Sorry, but I am clearly and obviously not a skilled programmer. – Christof De Backere Nov 08 '22 at 20:32
  • In VBA strings are stored in Unicode. This means that the string Hello is stored as 5 byte pairs, 0 'H' 0 'e' 0 'l' 0 'l' 0 'o' so passing string pointers is probably not what is needed for your api. Try reading this https://di-mgt.com.au/howto-convert-vba-unicode-to-utf8.html – freeflow Nov 08 '22 at 22:16

0 Answers0