If you know the max number of decimals then you could break apart the string into parts and sort on those individually. For example, if you know you'll only have a max of 4 decimals (separating 5 different numbers) then you could create 5 formulas each representing a piece of the string.
//Formula {@Num1} to isolate most significant number
local stringvar array splitString := split({table.string},'.');
if isnumeric(splitString[1]) then tonumber(splitString[1]) else 0
//...and the formula {@Num2} for second most significant number
local stringvar array splitString := split({table.string},'.');
if ubound(splitString)>1 and isnumeric(splitString[2])
then tonumber(splitString[2]) else 0
Now sort your report first by {@Num1}
and then by {@Num2}
etc.