strtok() is working correctly on WSL Ubuntu but not on Windows Mingw32

Question

The code below is giving me problems on Windows Mingw32 but not on WSL Ubuntu. The delimiter is (char)32 (space).

while (!feof(f)){
    if(!fgets(line, LINE_LEN, f) || !*line || !strcmp(line, "\n")) continue;
    word = strtok(line, &delim);
    printf("xd\n");
    while(word){
        //printf("%s\n",word);s
        add_item(h,word);
        word = strtok(NULL, &delim);
        wc++;
    }
    lc++;
}

I had tried debugging the code with CLion and the variable 'line' is correctly filled with given sentence that contain spaces, therefore strtok should not be returning null on the first iteration, yet it is. CLion Debug

Can you make this a [mcve] so we can see the full picture? `&delim` looks incorrect and `while (!feof(f))` is a poor choice for controlling the loop. — Retired Ninja, Dec 25 '22 at 14:14
@RetiredNinja hello &delim was indeed incorrect, delim was initialized as simple char ' ' and it didn't contain the \0 character, therefore it crashed. Thank you for your comment. — vatr, Dec 25 '22 at 14:21
Therein lies the folly of deciding that we only need see a fragment of the failing code, when you yourself don't know the problem. Rather than posting an answer that can only be arrived at through unseen information, you should edit the question to make it answerable. Or in this case probably simply delete the question. — Clifford, Dec 25 '22 at 15:08

Fe2O3 · Accepted Answer · 2022-12-25T15:18:13.607

Please replace that code with this:

// FILE *f presumed to be opened for reading
char line[ 1024 ];
int lc = 0, wc = 0;
while( fgets( line, sizeof line, f ) ) {
    for( char *wp = line; ( wp = strtok( wp, " \n" ) ) != NULL; wp = NULL ) {
        add_item( h, wp ); // unseen in this question.
        wc++;
    }
    lc++;
}

Empty lines can be safely loaded into line, and strtok() will deal with them correctly (ie: find no words.)

vatr · Answer 2 · 2022-12-25T14:53:38.480

0

Delim only contained the space character and did not contain the \0 character, therefore it crashed.

EDIT:

More detailed answer by @Joop Eggen below in comments to this answer:

strtok's second parameter is a string of delimiters, like " \t.", and hence a char delim = ' '; and &delim searches next chars till the string terminator '\0'.

edited Dec 25 '22 at 14:53

answered Dec 25 '22 at 14:23

vatr

25
3

1

A bit more please, like: `strtok`'s second parameter is a string of delimiters, like `" \t."`, and hence a `char delim = ' ';` and `&delim` searches next chars till the string terminator `'\0'`. – Joop Eggen Dec 25 '22 at 14:42

score 0 · Answer 3 · answered Dec 25 '22 at 15:41

The strtok function expects a C string as its second argument (a pointer to a null terminated array of char. delim is presumably defined as char delim = ' '; so &delim is a pointer to char but not a C string because there is a single character and no null terminator. The code has undefined behavior, it may appear to work on some platforms and not on others.

You should instead use a string " \n" or possibly " \t\r\n" as the delimiter string.

Furthermore, you should learn Why is “while( !feof(file) )” always wrong? . The loop should be change to: while (fgets(line, LINE_LEN, f)

Here is a modified version:

int parse_file(FILE *f, struct hash_table_t *h) {
    char line[LINE_LEN];
    const char *delim = " \t\r\n";
    int wc = 0, lc = 0;

    while (fgets(line, sizeof line, f)) {
        char *p = line;
        char *word;
        while ((word = strtok(p, delim)) != NULL) {
            add_item(h, word);
            wc++;
            p = NULL;
        }
        lc++;
    }
    return wc;
}

strtok() is working correctly on WSL Ubuntu but not on Windows Mingw32

3 Answers3