why is strtok() not working properly in my C program?

Question

I'm trying to develop a small console program that spits a string into separate words (i.e. tokens). Most of the program is working but I'm having a problem with the strtok() function. I have reviewed the code shown at How does strtok() split the string into tokens in C? and have based my own code somewhat on that. My problem lies in my makeWordList() function which is show below.

// Assumes a multi-word string was entered.
void makeWordList(char inString[], unsigned int wordCount, char * wordList[])    // Separate out words.
{
   unsigned int index = 0;
   const char * delimiter = " ";
   char * word;

   word = strtok(inString, delimiter);             // Get first word.

   while ((word != NULL) && (index < wordCount))
   {
      wordList[index++] = word;                    // Add word to list.
      word = strtok(inString, delimiter);          // Get next word.
   }  // end while()
}

In my case the strtok() function appears not to move along the (source) input string inString. When the program is run it produces the following output

./ex8_4
Enter string to be processed:
three word string
You entered |three word string|

There are 3 words in the string.

There are 15 letters in the string.

The word list is as follows.
three
three
three

From the output shown above it can readily be seen that strtok() successfully reads the first token (and terminates it with a "\000", according to codeblocks Watches panel) but not the subsequent tokens within inString. As I am using strtok() in a manner very similar to that shown in the code on the page linked to above, could someone please explain what it is that I am failing to understand ?

Please try to create a [mre] to show us, including how you call your `makeWordList` function. Hard-code the input string inside the `main` function. — Some programmer dude, Aug 29 '23 at 12:38
How would a caller of that function even know how many words were found? Is it supposed to store `NULL` in all elements of the `wordList` before calling that function? — Gerhardh, Aug 29 '23 at 12:41
From the `strtok()` manpage: "On the first call to strtok(), the string to be parsed should be specified in str. In each subsequent call that should parse the same string, str must be NULL." You're not doing the NULL part. — pmacfarlane, Aug 29 '23 at 12:41
`word = strtok(inString, delimiter); // Get next word.` That is not how `strtok` works. You must pass `NULL` on all calles except first one. — Gerhardh, Aug 29 '23 at 12:42

Vlad from Moscow · Answer 1 · 2023-08-29T13:00:38.247

In this while loop

   while ((word != NULL) && (index < wordCount))
   {
      wordList[index++] = word;                    // Add word to list.
      word = strtok(inString, delimiter);          // Get next word.
   }

change this statement

  word = strtok(inString, delimiter);          // Get next word.

to

  word = strtok(NULL, delimiter);          // Get next word.

Otherwise the function strtok tries to split the string anew from its beginning.

From the C Standard (7.23.5.8 The strtok function)

2 A sequence of calls to the strtok function breaks the string pointed to by s1 into a sequence of tokens, each of which is delimited by a character from the string pointed to by s2. The first call in the sequence has a non-null first argument; subsequent calls in the sequence have a null first argument. The separator string pointed to by s2 may be different from call to call.

Pay attention to that it would be better when the function will return the number of words in the passed string as for example

unsigned int makeWordList(char inString[], unsigned int wordCount, char * wordList[])    // Separate out words.
{
   unsigned int index = 0;

   //...

   return index; 
}

Bear in mind that the function strtok is not safe in multithreading environment because it uses a static variable. More safer to use alternative function strtok_s.

Saran Kumar · Answer 2 · 2023-08-29T13:10:57.100

The problem is that you are calling strtok() with the same string pointer (inString) in each iteration of the loop. This means that strtok() will only look for the next token starting from the end of the first token. In your case, the first token is "three", so strtok() will only find the next token starting from the end of "three". This is why the output of your program is "three three three".

    // Assumes a multi-word string was entered.
void makeWordList(char inString[], unsigned int wordCount, char * wordList[])    // Separate out words.
{
   unsigned int index = 0;
   const char * delimiter = " ";
   char * word;

   word = strtok(inString, delimiter);             // Get first word.

   while ((word != NULL) && (index < wordCount))
   {
      wordList[index++] = word;                    // Add word to list.

      // Get next word.
      word = strtok(NULL, delimiter);
   }  // end while()
}

why is strtok() not working properly in my C program?

2 Answers2