Strtok inside strtok not working with copy of original token - C

Question

I realize the title is confusing, couldn't think of a clearer way to word it. Basically, I am calling a strtok loop inside a strtok loop, but when the inner strtok function returns from runCommand, my first strtok loop stops. It simply exits the loop, even when there are other arguments following the first semicolon. When I don't call runCommand(), it works as expected, and parses through all my commands separated by semicolon.

The goal of this code is to parse a line of commands separated by semicolons, then parse command and command arguments to enter into execvp later. This is the only part I am having trouble with. Here it is:

void parseCommand(char *userLine) 
{
  if(strchr(userLine, ';'))
  {
    // Get first token
    token = strtok(userLine, ";");
    // Loop through all tokens
    while(token != NULL)
    {
      // Make a copy
      char *copy = malloc(strlen(token) + 1);
      strcpy(copy, token);
      runCommand(copy);
      free(copy);
      printf("process returned!\n");
      token = strtok(NULL, ";");
    }
  }
}
void runCommand(char *token)
{
  char *args[20]; 
  char **command = args;
  //Tokenize each command based on space

  char *temp = strtok(token, " \n");
  while (temp != NULL)
  {
    *command++ = temp;
    temp = strtok(NULL, " \n");
  }
  *command = NULL;
// code for fork and execvp here
}

Can someone explain why runCommand is screwing up my first function's parsing? I REALLY don't understand why it's not working with a copy of my original token. Probably simple, but I've looked at it too long?

"I have am calling a strtok loop inside a strtok loop" - can't do that. `strtok`'s design kind of sucks. — user2357112, Jan 12 '17 at 21:37
Yeah, you can't use `strtok` "recursively" like that - as soon as you pass a new, non-`NULL` argument, it completely forgets about its previous argument. You have to process your tokens breadth-first, not depth-first. — John Bode, Jan 12 '17 at 22:30
See documentation on [Tokenization: `strtok()`, `strtok_r()` and `strtok_s()`](http://stackoverflow.com/documentation/c/1990/strings/2557/tokenisation-strtok-strtok-r-and-strtok-s#t=20170113015701136593). — Jonathan Leffler, Jan 13 '17 at 01:59

Weather Vane · Accepted Answer · 2017-01-12T21:47:29.727

5

The function strtok is not reentrant. It remembers its current state, which is why you pass NULL for repeated calls without a segfault.

Consider using strtok_s or strtok_r (depending on implementation) which allows the caller to save the state. These can be used in a nested fashion.

edited Jan 12 '17 at 21:47

answered Jan 12 '17 at 21:42

Weather Vane

33,872
7
36
56

`strtok_r` is not standard, recommendation is to use `strtok_s`. – too honest for this site Jan 12 '17 at 22:16
THANK YOU!! Next time I'll look closer into the documentation. – Tee Jan 12 '17 at 22:25
@Olaf: If you work on Windows, `strtok_s()` is the one to use because it is available. If you work on Unix, `strtok_s()` is not available — POSIX rejected the functions in Annex K (see [Do you use the TR24731 'safe' functions](http://stackoverflow.com/questions/372980/do-you-use-the-tr-24731-safe-functions) for more information) and essentially no Unix system implements any of them — but `strtok_r()` is available. Fortunately, the two are functionally identical. Unless the platform is clear, stating unequivocally `strtok_s()` or `strtok_r()` is … misleading, at best. – Jonathan Leffler Jan 13 '17 at 02:03
@JonathanLeffler: I did not know that they refuded the whole annex. Interestingly some of the functions are nevertheless available, but as there seems to be no manpage for `strtok_s`, it indeed seems to be missing. Thanks for the info. (IIRC POSIX is based on C99 or even C90 still and the linked page is 3 years before C11. I wonder if a newer POSIX version might change that). – too honest for this site Jan 13 '17 at 11:33
@Olaf — AFAIK, the 2016 update to POSIX 2008 left the C version at C99. I've not verified that today though. – Jonathan Leffler Jan 13 '17 at 14:39

score 3 · Answer 2 · answered Jan 12 '17 at 21:42

strtok doesn't know about the context in which it's executing, it behaves somewhat globally.

Try using strtok_r, which allows you to specify a context so that multiple separate uses won't interfere with each other.

From the man page:

Different strings may be parsed concurrently using sequences of calls to strtok_r() that specify different saveptr arguments.

Strtok inside strtok not working with copy of original token - C

2 Answers2