VBA String Normalization (via WinAPI)

Question

I'm new to attempting to write code in VBA to use WinAPI functions. What encoding does the WinAPI Normalize() function work with? UTF-16 is what I would expect, but the following does not work. The number of characters seems like it's not calculated right, and then the attempt to actually create a normalized string will just crash Access.

'normFormEnum
'not random numbers, but from ...
'https://msdn.microsoft.com/en-us/library/windows/desktop/dd319094(v=vs.85).aspx
'for use in calling the Win API Function NormalizeString()
Public Enum normFormEnum
    normFOther = 0
    normFC = 1      'the W3C (Internet) required normalization format
    normFD = 2
    normFKC = 5
    normFKD = 6
End Enum

'https://msdn.microsoft.com/en-us/library/windows/desktop/dd319093(v=vs.85).aspx
Private Declare Function NormalizeString Lib "Normaliz" ( _
    ByVal normForm As normFormEnum, _
    ByVal lpSrcString As LongPtr, _
    ByVal cwSrcLength As Long, _
    ByRef lpDstString As LongPtr, _
    ByVal cwDstLength As Long _
    ) As Long

Public Function stringNormalize( _
    ByVal theString As String, _
    Optional ByVal normForm As normFormEnum = normFC _
    ) As String

    Dim nChars As Long
    Dim newString As String

    nChars = NormalizeString(normForm, StrPtr(theString), Len(theString), 0&, 0)

    'prefill the string buffer so it can be altered shortly...
    newString = String(nChars, " ")

Debug.Print nChars
'prints nChars, showing that it 3x the amount of characters.

'The following will crash the application....

'    NormalizeString normForm, StrPtr(theString), Len(theString), StrPtr(newString), nChars

    stringNormalize = newString

End Function

Stab in the dark: `Len` returns the number of *characters* in the string; have you tried `LenB` instead, which returns the number of *bytes* in the string? VBA strings use 2 bytes per character. — Mathieu Guindon, Jul 05 '17 at 15:39
@Mat'sMug: The Len() function returns the number of UTF-16 code units. The LenB() function will be double that and give an even worse result. — someprogrammer, Jul 05 '17 at 17:09
`Len` returns the number of ANSI characters in a string. That this matches UTF-16 code units is a coincidence. — Mathieu Guindon, Jul 05 '17 at 17:11
@Mat'sMug: Testing it here does not show that to be true. It shows it to give the number of code units. — someprogrammer, Jul 05 '17 at 17:13
[Len function on MSDN](https://msdn.microsoft.com/VBA/Language-Reference-VBA/articles/len-function) - *Returns a Long containing the number of characters in a string or the number of bytes required to store a variable.* - given a `String`, it's specified to return the number of characters. With examples. — Mathieu Guindon, Jul 05 '17 at 17:19

Florent B. · Accepted Answer · 2018-03-27T10:40:35.980

The function NormalizeString returns an estimated size in bytes when cwDstLength is 0, but you are using it as the number of characters.

So take half the result from the first call and truncate the buffer with the result from the second call:

Private Declare PtrSafe Function NormalizeString Lib "Normaliz" ( _
  ByVal normForm As Long, _
  ByVal lpSrcString As LongPtr, _
  ByVal cwSrcLength As Long, _
  ByVal lpDstString As LongPtr, _
  ByVal cwDstLength As Long _
) As Long

Public Enum NormalizationForm
  NormOther = 0
  NormC = 1
  NormD = 2
  NormKC = 5
  NormKD = 6
End Enum

Public Function NormalizeStr(source As String, ByVal normForm As NormalizationForm) As String
  Dim buffer As String, size As Long, i As Long

  For i = 1 To 5
    size = NormalizeString(normForm, StrPtr(source), Len(source), StrPtr(buffer), Len(buffer))

    If size >= 0 And size < Len(buffer) Then
      NormalizeStr = Left$(buffer, size)
      Exit Function
    End If

    buffer = String$(Abs(size) + 1, 0)
  Next

  Err.Raise 9, , "NormalizeString failed"
End Function

Public Sub Usage()
  Debug.Print NormalizeStr(ChrW(196), NormD)
  Debug.Print NormalizeStr("A" & ChrW(776), NormC)
End Sub

Thank you for your response. It doesn't work here. First, it gives a multiple of 3 for the size, not of 2. If it were a multiple of 2, I'd think it was bytes, but a multiple of 3 makes no sense to me. I've tried tweaking it a number of ways and it just crashes every time. — someprogrammer, Jul 13 '17 at 18:40
It works for me. The size from the first call is an estimated capacity in bytes allocated to fit the result for the worst case. It doesn't represent the size of the result. Since a unicode character is represented on 2 bytes, you need to divide this capacity by 2 to get an estimated number of characters. You'll get the real size from the second call. — Florent B., Jul 13 '17 at 18:53
Seems to work now. I added 2 to the buffer size instead of 1 (after division) and it works. — someprogrammer, Mar 13 '18 at 18:24

VBA String Normalization (via WinAPI)

1 Answers1

Linked