How to make "strtok function" work with more than one Token String at a time? Would function pointers solve this?

Question

strtok function uses a static variable for parsing the string into tokens. So this causes a conflict when multiple calls are done. Other than using threads how could I do the following: thx
- Can I use a function pointer to allocate the function at 2 different places? Would this make the static variable inside "strtok" allocate at 2 different places?

//breaking up first by Sentence and than by Word.
char phrase[] = "My dog has fleas.\nAnd he gave them to me."; 
char del1[]   = "\n";
char del2[]   = " ";
char *token1;
char *token2;


token1 = strtok( phrase, del1);

while( token1 != NULL )
{
    printf("Sentence:  %s",token1);

    token2 = strtok( token1, del2);
    while( token2 != NULL ){
        token2 = strtok( NULL, del2);
        printf("WORD:  %s",token2);
    }

    token1 = strtok( NULL, del1);
}

`My dog has flees`? What is a flee? Haha, fixed the grammar in your post. — Richard J. Ross III, Mar 31 '12 at 22:58

score 6 · Answer 1 · answered Mar 31 '12 at 22:05

6

use strtok_r() (re-entrant version).

answered Mar 31 '12 at 22:05

P.P

117,907
20
175
238

what header file holds this? Will this work is Visual c++ 6.0? – jdl Mar 31 '12 at 22:08
strtok_r is in `` header file. – Vincenzo Pii Mar 31 '12 at 22:12
In standard C, it's in string.h (cstring). You may have to lookup manuals for vc++. – P.P Mar 31 '12 at 22:12
(Visual c++ 6.0) I opened string.h... and I don't see strtok_r ... only strtok. – jdl Mar 31 '12 at 22:14
I am wanting to do this with out using MFC! – jdl Mar 31 '12 at 22:19
I see similar functions here: http://msdn.microsoft.com/en-us/library/2c8d19sb%28v=vs.110%29.aspx But I can't find their descriptions to know what they do. – P.P Mar 31 '12 at 22:29
2

@jdl: Here's a public domain `strtok_r()` implementation that should work fine in VC6: http://snipplr.com/view/16918/strtokr/ – Michael Burr Mar 31 '12 at 23:44
[Here's an example using strtok_r](https://stackoverflow.com/a/10349288/1275169). – P.P Nov 16 '18 at 20:20

William Morris · Accepted Answer · 2012-04-01T03:19:15.127

2

Instead of using strtok, perhaps use strsep. Note that I have extracted the nested loop into a function - nested loops suck!

EDITED: changed to use strsep directly

/* print each word in a string*/
static void print_words(char *s)
{
    while (s && *s) {
        char *t = strsep(&s, " ");
        printf("WORD:  %s\n", t);
    }
}

void loop(void)
{
    /* duplicate string in case it is read-only */
    char *phrase = strdup("My dog has flees.\nAnd he gave them to me.");

    while (phrase) {
        char *s = strsep(&phrase, "\n");
        printf("Sentence:  %s\n", s);
        print_words(s);
    }
}

edited Apr 01 '12 at 03:19

answered Mar 31 '12 at 22:38

William Morris

3,554
2
23
24

Keep in mind that `strsep()` has different behavior than `strtok()` when there is a run of delimiter characters (`strtok()` treats a run as a single field separator, `strsep()` treats a run of delimiters as several empty fields). Which behavior you might want depends on your needs. `strsep()` is also not available in VC++6 (which is what the OP apparently wants), though it has a pretty simple implementation (same problem as for for `strtok_r()`. – Michael Burr Apr 01 '12 at 00:06
1

In case the OP wants to use `strsep()` with VC++6, a public domain version can be found here: http://unixpapa.com/incnote/string.html – Michael Burr Apr 01 '12 at 00:10
I edited the code example to use `strsep` directly instead of via another function (which I had carelessly extracted from some other code without checking that it was needed). – William Morris Apr 01 '12 at 03:20
1

It's easy to get `strtok`-like behavior from `strsep`, but impossible to go the other way, which is why we made `strsep` behave the way it does. (Just add: `if (*phrase == '\0') continue;` or similar to skip empty "tokens".) Slogan: `strsep` ain't great, but it's better than `strtok`. :-) – torek Apr 01 '12 at 04:40
@chris: good point. I guess I was just considering the code as posted. – Michael Burr Apr 01 '12 at 07:44

score 1 · Answer 3 · answered Mar 31 '12 at 22:06

1

Use strtok_r(). It takes one more parameter - pointer which acts like a context.

answered Mar 31 '12 at 22:06

Vadym S. Khondar

1,428
2
12
19

score 1 · Answer 4 · answered Mar 31 '12 at 22:09

1

Is that C or C++?

If it is C++, you can use std::string instead of char * and std::string methods to achieve the same result as you get using strtok.

For example you could get benefits from find and substr methods.

answered Mar 31 '12 at 22:09

Vincenzo Pii

18,961
8
39
49

How to make "strtok function" work with more than one Token String at a time? Would function pointers solve this?

4 Answers4