VB dot NET ... How is it possible that this string doesn't get any shorter even though it clearly contains the text I am replacing?

Question

A client reported a problem where our software was "hanging". I eventually managed to find the piece of code responsible:

While strShortenedText.IndexOf("  ") > -1 'if there are two contiguous spaces
                                          'anywhere in this string ...
    strShortenedText = strShortenedText.Replace("  ", " ") '... replace each
                                                           'occurrence with a
                                                           'single string ...
End While '... and do this in a loop in order to handle situations with 3 or more
          'contiguous spaces

I'm afraid I won't be able to include the actual string here as it contains data covered by GDPR legislation, but I am just looking for an explanation as to how this could possibly have gone into an infinite loop?

What I find is that:

1. strShortenedText.IndexOf(" ") returns a value greater than -1, so somewhere in this string there are two contiguous spaces (I made sure that these spaces are both 0x20 i.e. char(32))
2. So if strShortenedText contains " " (2 contiguous spaces) anywhere, and I replace " " (2 contiguous spaces) with " " (a single space) then, surely (??????), strShortenedText after the .Replace will be shorter than it was before the replace?

Instead, I find that the REPLACE leaves the string unchanged, and no matter how often the code loops through the statement, the variable's length never changes.

How might this be possible? And what could I possibly do to handle this scenario?

Answer 1

The problem is that IndexOf and Replace use different ways of finding text by default. That means that if you have a string containing U+200C (zero-width non-joiner) for example, IndexOf will skip over it and find the two spaces each side, but Replace won't.

You need to make IndexOf and Replace use a consistent way of looking for text. The simplest to understand (but potentially not the most desirable) option is to use an ordinal check:

Module Program
    Sub Main(args As String())
        Console.WriteLine(ReplaceMultipleSpaces("x " & ChrW(&H200C) & " y"))
    End Sub

    Function ReplaceMultipleSpaces(text As String)
        While text.IndexOf("  ", StringComparison.Ordinal) > -1
            text = text.Replace("  ", " ", StringComparison.Ordinal)
        End While
        Return text
    End Function
End Module

(Using String.Contains instead of String.IndexOf is probably preferable in general - but comes with the same requirement to specify the kind of check you want to perform, if you don't want to just use the default.)

Answer 2

... what could I possibly do to handle this scenario?

Exit the loop once you didn't make any replacements

Dim lastLength = strShortenedText.Length
While strShortenedText.IndexOf("  ") > -1 
    strShortenedText = strShortenedText.Replace("  ", " ")
    If lastLength = strShortenedText.Length Then Exit While
    lastLength = strShortenedText.Length
End While

Or switch to Do...While,

Dim lastLength As Integer
Do
    lastLength = strShortenedText.Length
    strShortenedText = strShortenedText.Replace("  ", " ")
Loop While lastLength > strShortenedText.Length

If you want to actually handle the special character, this may work for you. Use a function to combine all your chars into strings which you iterate over to perform replacements. It will also allow you to add additional characters to be replaced. The combinations are in pairs, and all possible combinations, so it may be a little inefficient i.e. AA, AB, BA, BB

Private Iterator Function getPairsOfChars(chars As IEnumerable(Of Char)) As IEnumerable(Of String)
    For i = 0 To chars.Count() - 1
        For j = 0 To chars.Count() - 1
            Yield New String({chars(i), chars(j)})
        Next
    Next
End Function

Public Sub test()
    Dim strShortenedText = $"  {ChrW(173)}  s   s   s  {ChrW(173)}{ChrW(173)}    s {ChrW(173)}{ChrW(173)}{ChrW(173)}    s {ChrW(173)}s    s{ChrW(173)}s "
    Dim lastLength As Integer
    Dim pairs = getPairsOfChars({ChrW(32), ChrW(173)})
    Do
        lastLength = strShortenedText.Length
        For Each pair In pairs
            strShortenedText = strShortenedText.Replace(pair, " ")
        Next
    Loop While lastLength > strShortenedText.Length
End Sub

Answer 3

As mentioned in the Comments, the cause of the problem was a non-printable character (ASCII code 173) between the two spaces. Apparently the .IndexOf method does NOT see this character, whereas the .Replace method does. As a result, the .IndexOf method finds a position where it thinks there are two contiguous spaces, but the .Replace method then cannot replace that as the actual text in that location is not two ASCII characters 32, but ASCII character 32, followed by ASCII character 173, followed by a second ASCII character 32.

The solution then is to code around that possibility:

    nDoubleSpace = strShortenedText.IndexOf("  ") 'Okay - we found a situation where there may be non-printing characters
    '                                              between two spaces, where this IndexOf method returns a value > -1 ....
    '                                              And this caused an infinite loop as the Replace did nothing.
    While nDoubleSpace > -1
        If strShortenedText.Substring(nDoubleSpace, 2) <> "  " Then
            'The IndexOf found a location with two spaces separated by non-printing characters (for example, character 173),
            'which means a .Replace won't work. Sadly this software targets .NET Framework, not .NET Core so we don't have
            'the option to use a StringComparison override in the .Replace. This is the next best thing we can do...
            'Basically, we know that the location we found with the .IndexOf STARTS and ENDS with a space. So instead of
            'doing a .Replace, we just concatenate everything before the FIRST space with everything
            'from the SECOND space onward.
            Dim nNextSpace As Integer
            nNextSpace = 1
            'Find the position of the second space
            While strShortenedText.Substring(nDoubleSpace + nNextSpace, 1) <> " "
                nNextSpace = nNextSpace + 1
            End While
            '                  everything before the first space             everything from the second space onward
            strShortenedText = strShortenedText.Substring(0, nDoubleSpace) + strShortenedText.Substring(nDoubleSpace + nNextSpace)
        Else
            'Now we know that the IndexOf and the Substring are in agreement, we can be happy that this
            'simple Replace will do the job
            strShortenedText = strShortenedText.Replace("  ", " ")
        End If

        'And check again to handle more than 2 contiguous spaces
        nDoubleSpace = strShortenedText.IndexOf("  ")
    End While

Yeah it's ugly. But it works

Answer 4

Maybe some Regex?

    Dim strShortenedText As String = "More     " & Chr(173) & "  or less"
    'before the or(|) there is a soft hyphen
    Dim regx As New System.Text.RegularExpressions.Regex("(­|\s)\s+", System.Text.RegularExpressions.RegexOptions.Compiled)
    strShortenedText = regx.Replace(strShortenedText, " ")

Answer 5

If there are other non-standard 'space' characters this approach seems the easiest to maintain.

    ' https://unicode-explorer.com/articles/space-characters
    ' 
    Dim spcChars() As Char = {" "c, ControlChars.Tab, ChrW(173), ChrW(&H2000), ChrW(8192), ChrW(8193), ChrW(8194), ChrW(8195)}

    Dim strShortenedText As String = "More     " & Chr(173) & Chr(173) & ControlChars.Tab & "  or less"
    Dim splt() As String = strShortenedText.Split(spcChars, StringSplitOptions.RemoveEmptyEntries)
    strShortenedText = String.Join(" "c, splt)

Answer 6

I can't test this, since I don't have your string, but if you're looking for a double-space and find one, then Replace can't find it, how about just shortening the string?

Dim spaceIndex As Integer = strShortenedText.IndexOf("  ")
While spaceIndex > -1 'if there are two contiguous spaces anywhere in this string ...
    strShortenedText = Left(strShortenedText, spaceIndex + 1 + Mid(strShortenedText, spaceIndex + 2) '... remove the first space found
    ' Note that IndexOf is zero-based and Left and Mid are 1-based
    spaceIndex = strShortenedText.IndexOf("  ")
End While

I'm not sure what this will do with your 173 character, but I think it's worth a try, since it is a lot shorter if it works.