No, it can not be done with regular expressions "directly". And here you can read why.
Anyway, for a solution using regular expressions (a lot of code, but depending of your data length it can be faster or not, you will need to try)
Dim dicEncode
set dicEncode = WScript.CreateObject("Scripting.Dictionary")
Dim encodeRE
Set encodeRE = New RegExp
With encodeRE
.Pattern = "\{[^{}]*\}"
.Global = True
.IgnoreCase = True
End With
Dim decodeRE
Set decodeRE = New RegExp
With decodeRE
.Pattern = "\x00(K[0-9]+)\x00"
.Global = True
.IgnoreCase = True
End With
Function encodeFunction(matchString, position, fullString)
Dim key
key = "K" & CStr(dicEncode.Count)
dicEncode.Add key , matchString
encodeFunction = Chr(0) & key & Chr(0)
End Function
Function decodeFunction(matchString, key, position, fullString)
decodeFunction = dicEncode.Item(key)
End Function
Dim originalString
originalString = "{abc.def{gh.ijk}l.m}n.o.p{q.r{s{t{u.v}}}w}.x"
Dim encodedString, workBuffer
encodedString = originalString
Do
workBuffer = encodedString
encodedString = encodeRE.Replace(encodedString,GetRef("encodeFunction"))
Loop While encodedString <> workBuffer
encodedString = Replace(encodedString, ".", Chr(0))
Do
workBuffer = encodedString
encodedString = decodeRE.Replace(encodedString,GetRef("decodeFunction"))
Loop While encodedString <> workBuffer
Dim aElements, element
aElements = Split(encodedString, Chr(0))
WScript.Echo originalString
For Each element In aElements
WScript.Echo element
Next
All this code just uses regular expressions to find the pairing curly brackets in the string, replacing them and its enclosed data with a key indicator that is stored in a dictionary. When all the "enclosed" data is removed from the string, the remaining dots (your split points) are replaced with a new character (that will later be used to split the string) and then the string is reconstructed. All the "enclosed" dots has been protected and the split can be done over the string using the new character ( Chr(0) in the code).
It is similar to the dictionary creation of a statistical compressor but without any statistics and no compression, of course.
But only useful with long strings. If not, your original approach is way better.
EDITED to adapt to comments
For a better performing code, based on the OP original approach. No exotic regular expresions. Just reduced string concatenation and unnecesary checks removed.
Function mySplit(originalString)
Dim changedString, currentPoint, currentChar, stringEnd, level
changedString = originalString
stringEnd = Len(originalString)
level = 0
For currentPoint = 1 To stringEnd
currentChar = Mid(originalString, currentPoint, 1)
If currentChar = "{" Then
level = level + 1
ElseIf currentChar = "}" Then
If level > 0 Then
level = level - 1
End If
ElseIf level = 0 Then
If currentChar = "." Then
changedString = Left(changedString,currentPoint-1) & Chr(0) & Right(changedString,stringEnd-currentPoint)
End If
End If
Next
mySplit = split( changedString, Chr(0) )
End Function