Strtok usage with NULL

Question

I found this example for using strtok() on http://www.cplusplus.com/reference/cstring/strtok/, and don't understand how they are getting from token to token through "pch = strtok (NULL, " ,.-");"

Here is the code:

/* strtok example */
#include <stdio.h>
#include <string.h>

int main ()
{
  char str[] ="- This, a sample string.";
  char * pch;
  printf ("Splitting string \"%s\" into tokens:\n",str);
  pch = strtok (str," ,.-");
  while (pch != NULL)
  {
    printf ("%s\n",pch);
    pch = strtok (NULL, " ,.-");
  }
  return 0;
}

Here is the output:

Splitting string "- This, a sample string." into tokens:
This
a
sample
string

Read [the manual](http://linux.die.net/man/3/strtok). It explains that very explicitly. If after reading that you still don't understand then please describe what specifically you don't understand about what it says. — kaylum, Nov 16 '15 at 03:36
Don't use `strtok()` in library code. If you need to tokenize, consider `strtok_r()` on Unix and `strtok_s()` on Windows. Or use other techniques to tokenize your data. Functions such as `strcspn()`, `strspn()` and `strpbrk()` are available everywhere (they're part of the C89/C90 standard), and can be very useful. — Jonathan Leffler, Nov 16 '15 at 03:41
You might look at [What are the differences between `strtok()` and `strsep()`?](https://stackoverflow.com/questions/7218625) and [Nested `strtok()` function problem in C](https://stackoverflow.com/questions/4693884) and [How to use `strtok()`?](https://stackoverflow.com/questions/18927793) They all have relevant information in the answers. — Jonathan Leffler, Nov 16 '15 at 03:49

elixenide · Accepted Answer · 2015-11-16T03:44:37.810

strtok keeps an internal state via static variables so that you can work your way through a string in multiple calls. After you call strtok(str, ...), subsequent calls to strtok(NULL, ...) will get subsequent tokens from str. In other words, the first call gets the first token. The subsequent calls reuse the same string by passing NULL and this get the subsequent tokens.

As explained in the manual,

strtok() function parses a string into a sequence of tokens. On the first call to strtok() the string to be parsed should be specified in str. In each subsequent call that should parse the same string, str should be NULL.

`keeps internal state via static variables` This is what I needed. Thank you. — WISERDIVISOR, Mar 10 '23 at 15:35

score 1 · Answer 2 · answered Nov 16 '15 at 03:41

The strtok function keeps state in static variables, which are retained between function calls.

The first time you call strtok with the first argument non-NULL, it knows that this is the first time you're trying to parse this string, so it starts fresh and remembers where it stopped. When you call it again with NULL for the first argument, it uses the static variable to pick up where it left off.

Because of this, you can't use strtok to parse two separate strings interchangeably, nor can you use it in a multithreaded application. For that, the strtok_r function is more appropriate. Rather than using a static variable, strtok_r takes a third argument which is a "save pointer" it be used to keep track of where it is.

One other thing to keep in mind is that strtok and strtok_r modify the string they're parsing. If you don't want this, either create a copy of the string or use a different set of functions.

Note that Windows has `strtok_s()` instead of `strtok_r()` — but apart from the spelling they are equivalent. — Jonathan Leffler, Nov 16 '15 at 03:50

Strtok usage with NULL

2 Answers2