Lifetime of tokens return from strtok

Question

Having read the documentation from man7.org and some questions on StackOverflow, I'm still having trouble understanding who manages the lifetime of tokens returned from the strtok function. Most of the examples I've seen show the following usage for the strtok function:

char *token = NULL;

token = strtok(buffer, " ");

while (token != NULL) {
    token = strtok(NULL, " ");
}

I understand that strtok returns a pointer to the next token, and NULL when there are no more tokens in the string. However, we're not preallocating any storage (on the stack, or using malloc) for these tokens. How does strtok allocate storage for each token? Will I have to call free on each token to prevent a memory leak? Can I put the tokens into an array without using strcpy or will they go out of scope and be popped off the stack, leaving me with an array of dangling pointers?

Thanks!

It returns a pointer into `buffer`: no memory is allocated. `strtok` modifies `buffer` inserting characters to delimit the next token. — hmjd, Oct 09 '14 at 16:06
Just a quick note: on Stack Overflow, "hello" and "thanks" are not needed and should be removed. — Léo Lam, Oct 09 '14 at 16:09

score 0 · Accepted Answer · answered Oct 09 '14 at 16:12

For questions like these, reading the source of an strtok() implementation can be very instructive.

In short:

It returns pointers into buffer, i.e. your original string.
Yes, tokens are replaced by string terminators, which is why strtok() is so scary and not thread-safe.

Lifetime of tokens return from strtok

1 Answers1