strtok() - Why you have to pass the NULL pointer in order to get the next token in the string?

Question

This is the explanation of strtok().

#include <string.h>
char* strtok( char* s1, 
              const char* s2 );*

The first call to strtok() returns a pointer to the first token in the string pointed to by s1. Subsequent calls to strtok() must pass a NULL pointer as the first argument, in order to get the next token in the string.

But I don't know, why you have to pass the NULL pointer in order to get the next token in the string. I searched about 15 minutes, but didn't find an explanation in the internet.

If code did not pass `NULL`, what should code pass to continue tokenizing the same string? — chux - Reinstate Monica, Apr 14 '15 at 22:43
I guess it's just so you can opt out of tokenizing the whole string at once while not keeping track of where to start the next token yourself. If this was designed in C++ right now, it would most likely be different. — chris, Apr 14 '15 at 22:43
You did not see http://www.cplusplus.com/reference/cstring/strtok/ in your search? — Jongware, Apr 14 '15 at 22:43
Becaus that's the way it was designed 40 years ago. Poorly. Single-threaded and single-usage (not nestable), instead of providing a context object. — user207421, Apr 14 '15 at 22:45

score 43 · Accepted Answer · edited Feb 10 '19 at 07:06

strtok() keeps some data inside of itself by using static variables. This way, strtok() can continue searching from the point it left off at during the previous call. To signal strtok() that you want to keep searching the same string, you pass a NULL pointer as its first argument. strtok() checks whether the first argument is NULL and if it is, it uses its currently stored data. If the first parameter is not null, it is treated as a new search and all internal data is reset.

Maybe the best thing you can do is to search for an actual implementation of the strtok() function. I've found one small enough to post it here, so you get an idea of how to handle this NULL parameter:

/* Copyright (c) Microsoft Corporation. All rights reserved. */

#include <string.h>

/* ISO/IEC 9899 7.11.5.8 strtok. DEPRECATED.
 * Split string into tokens, and return one at a time while retaining state
 * internally.
 *
 * WARNING: Only one set of state is held and this means that the
 * WARNING: function is not thread-safe nor safe for multiple uses within
 * WARNING: one thread.
 *
 * NOTE: No library may call this function.
 */

char * __cdecl strtok(char *s1, const char *delimit)
{
    static char *lastToken = NULL; /* UNSAFE SHARED STATE! */
    char *tmp;

    /* Skip leading delimiters if new string. */
    if ( s1 == NULL ) {
        s1 = lastToken;
        if (s1 == NULL)         /* End of story? */
            return NULL;
    } else {
        s1 += strspn(s1, delimit);
    }

    /* Find end of segment */
    tmp = strpbrk(s1, delimit);
    if (tmp) {
        /* Found another delimiter, split string and save state. */
        *tmp = '\0';
        lastToken = tmp + 1;
    } else {
        /* Last segment, remember that. */
        lastToken = NULL;
    }

    return s1;
}

* WARNING: Only one set of state is held and this means that the * WARNING: function is not thread-safe nor safe for multiple uses within * WARNING: one thread. What does this mean? — Minh Tran, Oct 16 '15 at 22:17
@MinhTran It means this function should not be used in multi thread call because static variable maybe changed by other thread. See `/* UNSAFE SHARED STATE! */` — thangdc94, Apr 02 '17 at 15:07

score 13 · Answer 2 · answered Apr 14 '15 at 22:41

13

If you pass a non-NULL value, you are asking it to start tokenizing a different string.

If you pass a NULL value, you are asking to continue tokenizing the same thing as before.

answered Apr 14 '15 at 22:41

StilesCrisis

15,972
4
39
62

1

But, how does strtok know, that it is the same string? Now you pass NULL, but that is not the address of the string. – phez1 Apr 14 '15 at 22:45
7

@ezph1, It handles only one string at a time and keeps internal state, which causes all sorts of problems. – chris Apr 14 '15 at 22:46

strtok() - Why you have to pass the NULL pointer in order to get the next token in the string?

2 Answers2