strtok_s behaviour with consecutive delimiters

Question

I'm parsing 3 values in parallel which are separated with a specific separator.

token1 = strtok_s(str1, separator, &nextToken1);
token2 = strtok_s(str2, separator, &nextToken2);
token3 = strtok_s(str3, separator, &nextToken3);

while ((token1 != NULL) && (token2 != NULL) && (token3 != NULL))
{
    //...
    token1 = strtok_s(NULL, separator, &nextToken1);
    token2 = strtok_s(NULL, separator, &nextToken2);
    token3 = strtok_s(NULL, separator, &nextToken3);
}

Suppose '-' is my separator. The behaviour is that a string with no consecutive separators:

1-2-3-45

would effectively result in each of these parts:

However, a string with two consecutive separators:

1-2--3-45

will not yield a 0 length string, that one is skipped so that the result is:

and not

What workaround or strategy would be better suited to obtain all the actual parts, including the 0-length ones? I'd like to avoid re-implementing strtok_s, if possible.

strtok() is evil because it modifies the input string. Other than that, I'm curious about the answer to this. — Daniel Kamil Kozar, Jan 23 '12 at 20:07
In my case, strtok can modify the string all it wants, I'm keeping the original and working on a copy. — MPelletier, Jan 23 '12 at 20:08
I got into Microsoft's strtok_s.inl and saw that indeed this is the expected behaviour because there is a skip of leading delimiters to the first "non-delimiter" character. Hmm... — MPelletier, Jan 23 '12 at 20:30

drrlvn · Accepted Answer · 2012-01-23T20:19:11.507

6

Unfortunately, strtok() ignores empty tokens. Even though you said you wish to avoid doing that, there is no other way but to parse it yourself, using for example strchr() to find the next delimiter and then copying the token to a temporary variable for processing. This way you can handle empty tokens whichever way you please.

edited Jan 23 '12 at 20:19

answered Jan 23 '12 at 20:13

drrlvn

8,189
2
43
57

Jonathan Wood · Answer 2 · 2012-01-23T20:18:11.887

Yes, that's the way this function works. It's more appropriate for tasks like parsing words where multiple whitespace characters should not be treated as empty words.

I've done a lot of parsing. I would simply write my own parser here, where the code examines one character at a time. It's not that difficult and you can make it behave exactly how you need. As an example, I've posted some C++ code to parse a CSV file in my article Reading and Writing CSV Files in MFC

strtok_s behaviour with consecutive delimiters

2 Answers2

Linked