Strtok in C++ abnormal behaviour

Question

I am trying to use strtok in C++ to get tokens of a string. But, I see that in one ur of 5 run the tokens being returned by function are incorrect. Can someone pls, suggest what can be the problem?

Sample code reproducing the issue I am facing:

#include<iostream>
#include<vector>
#include<cstring>

using namespace std;
#define DEBUG(x) cout<<x<<endl;


void split(const string &s, const char* delim, vector<string> & v)
{
        DEBUG("Input string to split:"<<s);

        // to avoid modifying original string first duplicate the original string and return a char pointer then free the memory
        char * dup = strdup(s.c_str());
        DEBUG("dup is:"<<dup);
        int i=0;
        char* token = strtok(dup,delim);

        while(token != NULL)
        {
                DEBUG("token is:"<<string(token));
                v.push_back(string(token));
                // the call is treated as a subsequent calls to strtok:
                // the function continues from where it left in previous invocation
                token = strtok(NULL,delim);
        }
        free(dup);
}

int main()
{
        string a ="MOVC R1,R1,#434";

        vector<string> tokens;
        char delims[] = {' ',','};
        split(a,delims,tokens);
        return 0;
}

Sample output:

mayank@Mayank:~/Documents/practice$ ./a.out 
Input string to split:MOVC R1,R1,#434
dup is:MOVC R1,R1,#434
token is:MOVC
token is:R1
token is:R1
token is:#434

mayank@Mayank:~/Documents/practice$ ./a.out 
Input string to split:MOVC R1,R1,#434
dup is:MOVC R1,R1,#434
token is:MO
token is:C
token is:R1
token is:R1
token is:#434

As you can see in second run the tokens created are MO C R1 R1 #434 and not MOVC R1 R1 #434

I tried checking library code too but not able to figure out the mistake. Please help.

EDIT1: My gcc version is: gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12)

Can you please suggest any other method? I want to split string using multiple delimiters. Also, I don't want to use boost library. — Mayank Jain, Oct 30 '16 at 19:10
Here's some direction: http://stackoverflow.com/questions/24504582/how-to-test-whether-stringstream-operator-has-parsed-a-bad-type-and-skip-it — πάντα ῥεῖ, Oct 30 '16 at 19:15
This isn't the problem, but don't use `std::endl` unless you need the extra stuff that it does. `'\n'` ends a line. — Pete Becker, Oct 30 '16 at 19:18
@PeteBecker - I used `endl` because in case program ends abnormally with Segmentation fault or something there may be chance that buffer was not cleared. So to clear buffer and be on safe side I used endl — Mayank Jain, Oct 30 '16 at 19:22
@krzaq why did you delete your answer? it proposes an alternate full C++ solution to `strtok` which is C and requires `strdup`, etc... bad. and you had figured out the problem too... — Jean-François Fabre, Oct 30 '16 at 19:32
Can't speak for krzaq, @Jean-FrançoisFabre, but I didn't post anything because you beat me to the main bug and a google of "split string multiple delimiters c++" turned up half a dozen good SO answers already. Why pollute with more? — user4581301, Oct 30 '16 at 20:06
agreed. Also agreed that my answer is more useful to C users than to C++ users which should use "modern" split methods. — Jean-François Fabre, Oct 30 '16 at 20:21

Jean-François Fabre · Accepted Answer · 2016-10-30T19:16:39.653

9

char delims[] = {' ',','};

should be

char delims[] = " ,";

You're passing a list of chars instead of a char * bearing the list of delimiters to use, hence unexpected behaviour because strtok expects a 0-terminated string. In your case, strtok goes "in the woods" and tokenizes with anything after the declared array.

edited Oct 30 '16 at 19:16

answered Oct 30 '16 at 19:10

Jean-François Fabre

137,073
23
153
219

Strtok in C++ abnormal behaviour

1 Answers1