strtok_r throwing in random new lines

Question

Hi everyone I've got a problem with strtok_r throwing in random new lines, and was wondering if anyone could show me why it would be doing that? From my understanding, it is printing a new line when the input has a new line at the end, but this doesn't happen every time and I'm not sure why. My motivation is to have an input string of multiple words, and for strtok_r to store the words individually into an array (storage).

char delimit[] = " \t\r\n\v\f";
char *tempword; //temporary word until it is stored into the temp array
for (int r = 0; r < line_count; r++) {
    int counting = 0; //location of where tempword is stored in temp[counting]
    tempword = strtok_r(ptrarray[r], delimit, &ptrarray[r]);
    while (tempword != NULL && strcmp(tempword,"\n") != 0 && strcmp(tempword, "\0") != 0) {
        printf("temp: %s\n", tempword);
        storage[r][counting] = strdup(tempword); 
        tempword = strtok_r(NULL, " ", &ptrarray[r]);
        counting++;   
    }
    storage[r][word_count] = NULL; //last argument = NULL for execvp function
}

Why do you have `ptrarray[r]` ***twice*** in your call to `strtok_r()`? Have you looked at any [examples of how to use `strtok_r()`](https://stackoverflow.com/questions/15961253/c-correct-usage-of-strtok-r)? — Andrew Henle, May 02 '21 at 00:10
On your first call to `strtok_r()`, you pass a delimiter string that includes several whitespace characters, including the newline. The token, if any, parsed by that call will not contain a newline. On the other hand, the calls inside the loop all pass a delimiter string containing only a space character. This is valid, but not necessarily what you actually want. Other than that, we probably need to see a [mre] to determine what's going on. — John Bollinger, May 02 '21 at 00:39
Note also that although it is not technically wrong to have `strtok_r()` re-use `ptrarray[r]` for forwarding state between calls, it is fishy, and it might cause issues elsewhere. You can surely spare the space for a single independent pointer to be used for the purpose. — John Bollinger, May 02 '21 at 00:42

score 1 · Accepted Answer · answered May 05 '21 at 07:03

The third argument to strtok_r should be the address of a char * to store the internal state for the next call to strtok_r with a NULL initial pointer to return the next token. You pass the address of the string pointer you are trying to parse, this is incorrect. You should instead pass the address of a separate variable.

Furthermore strtok_r will not return a pointer to the newline nor a pointer to an empty string, so the extra tests are redundant.

Finally, you should pass the same string of delimiters for all tokens.

Here is a modified version:

char delimiters[] = " \t\r\n\v\f";
for (int r = 0; r < line_count; r++) {
    char *tempword; //temporary word until it is stored into the temp array
    char *state;
    int counting = 0; //location of where tempword is stored in temp[counting]
    tempword = strtok_r(ptrarray[r], delimiters, &state);
    while (tempword != NULL) {
        printf("temp: %s\n", tempword);
        storage[r][counting] = strdup(tempword); 
        tempword = strtok_r(NULL, delimiters, &state);
        counting++;   
    }
    storage[r][word_count] = NULL; //last argument = NULL for execvp function
}

strtok_r throwing in random new lines

1 Answers1