trouble using strtok to parse CSV file with date/time in c++

Question

I have a CSV file with dates (that I converted from excel) but I'm having trouble parsing them using the strtok function. When I delete the column with dates from the CSV file it works fine, but I get an error otherwise. I've been playing around with it for hours but the error(s) won't go away. I'd greatly appreciate any help/advice! I'm a programming beginner. Here is what I have (bolded part is the part I took out for the program to work, without dates in the CSV file):

void parseFile (ifstream &myfile) {

string line;
int n = 1;

while ( myfile.good() ) {

    //storing each line into char array
    getline(myfile,line);
    char data[100];
    for (int g = 0; g <= line.size(); g++) {
        data[g] = line[g];
    }

    //using strtok to read char array
    char * tok;
    tok = strtok (data,",");

    while (tok != NULL) {

        temp.patientID = atol(tok);
        tok = strtok(NULL, ",");

        temp.result = atof(tok);
        tok = strtok(NULL, ",");
        tok = strtok(NULL, "/");
        tok = strtok(NULL, "/");

        temp.date = atof(tok);
        tok = strtok(NULL, " ");
        tok = strtok(NULL, ":");
        tok = strtok(NULL, ":");

        temp.time = atof(tok);
        tok = strtok(NULL, ",");

        //getting each component and storing into map that stores parsed info, mymap
        mymap.insert (pair<int,testInfo>(n,temp));
        n++;
    }
}

}

My CSV file looks something like this:

1000856,0,28/09/2014 02:34:37
1002259,0.008,15/09/2014 23:14:11
1002259,0.002,18/09/2014 10:44:18
1002259,0.005,18/09/2014 16:54:52
1003348,0.038,20/03/2015 12:50:46

You might find the boost libraries useful for this kind of thing - particularly this http://stackoverflow.com/questions/5734304/c-boost-split-string as well as the boost `date_time` libs — mathematician1975, Sep 22 '15 at 22:16

score 0 · Answer 1 · answered Sep 22 '15 at 22:30

You are searching for one more comma then needed, which means your token after temp.result = atof(tok); tok = strtok(NULL, ","); points to the end of the string. Remove that line and I think it should do what you want. The segfault occurs when you try and convert the NULL output from strtok to a float.

user4581301 · Accepted Answer · 2015-09-22T23:10:36.633

Looking at this line:

1000856,0,28/09/2014 02:34:37

I see the following delimiters: ,,// ::

You strtok for ,,,// ::,

One too many ','s up front.

The line will parse to: "1000856" "0" "28/09/2014 02:34:37" NULL NULL NULL NULL NULL NULL and those NULLs will play absolute havoc on the atofs.

In addition, temp.date = atof(tok); will not give you a date. It would give you part of the date. The month I think. temp.time = atof(tok); is just going to give you seconds.

For this task a better option to strtok is stringstream and getline(stream &, string &, char). This guy lets you set the delimiter and so stuff like this:

std::stringstream linestream(line)
std::string tok;
if (getline(linestream, tok, ','))
{ // got a token
    try
    { 
        temp.patientID = std::stol(tok);
    }
    catch (...)
    {
        // handle bad input number
    }
}

I also replaced atol with std::stol because with it you can stay in string land and catch an exception if the number doesn't convert. atol fails silently when it can't convert.

If you are not allowed to use streams, look at sscanf. For regularly formatted data like you have it's the bomb compared to strtok.

By the way, good on you for figuring out you need to copy the data from line before passing it into strtok.

The above is easy to write and can be quite slow performing, but should be fine here.

A few things to look at here that are rendered obsolete by my above suggestion:

char data[100];
for (int g = 0; g <= line.size(); g++) {
    data[g] = line[g];
}

g <= line.size() will always read one past the end of the string. Great because it preserves the NULL, but not entirely kosher. I'm not sure if C++ guarantees there will be a NULL there. I probably should buy and read a copy of the standard one of these days. In addition to testing for g <= line.size(), you should also test for g < 100 to prevent overrunning the the char array.

Inside the while loop, you have nothing to prevent destruction if strtok returns NULL because of a poorly formatted line.

if mymap is a std::map, you can simplify

mymap.insert (pair<int,testInfo>(n,temp));

to

mymap[n] = temp;

trouble using strtok to parse CSV file with date/time in c++

2 Answers2