bad pointer: expression can not be evaluated while parsing tokens

Question

While parsing tokens , my all token pointers become bad and when i check in debugger it says expression can not evaluated. Error:CX0030 :Error: expression can not be evaluated and the token has value 0x00000000

I am missing something here, i tried enough but could not correct it. i know its something to do with pointer declration like const etc, but making changes does not help.

Windows + visual studio 2010 and files being parsed is unicode UTF16

code snippet ------

    const wchar_t* const DELIMITER = L"\"";
wchar_t buf[MAX_CHARS_PER_LINE];
fin.getline(buf, MAX_CHARS_PER_LINE);
wchar_t* token[MAX_TOKENS_PER_LINE] = {};
token[0] = wcstok(buf, DELIMITER);
if (token[0]) // zero if line is blank
    {
    int n = 0; 
         for (n = 0; n < MAX_TOKENS_PER_LINE; n++)   
        {
            token[n] = wcstok(0, DELIMITER); // subsequent tokens   --> error ,pointer becomes bad

            if (!token[n]) break; // no more tokens  -------------> code does not go beyond this

buf has entire file (this is another problem that when i try to read line it gets entire file in it), can someone help me point what mistake i did here?

If this is C++, I don't recommend using this tokenizing function. — chris, Dec 05 '13 at 09:59
@Exceptyon , i did not understand what you mean, do you mean i am missing declaration? or you mean i should not use the token as array? — NxC, Dec 05 '13 at 10:05
@chris , its C++ , i have rest of my code (almost 2k lines in C++ so cant change it now) , do you have any alternative function that i can work out? — NxC, Dec 05 '13 at 10:06
@NeileshC, http://stackoverflow.com/questions/236129/how-to-split-a-string-in-c — chris, Dec 05 '13 at 10:08

score 0 · Answer 1 · answered Dec 05 '13 at 10:00

0

You are losing the first token because n starts at 0. For the rest the only thing I can think of that this code can run from multiple threads so that the static buffer inside wcstok gets overwritten by subsequent calls of wcstok(buf, DELIMITER). This works fine (from singlethreaded app):

char buf[] = "1;2;3;4";

char *tok[10] ={};

tok[0] = strtok(buf, ";");

if (tok[0])
{
    int n = 1;

    while ((tok[n] = strtok(0, ";")) != 0)
    {
        n++;
    }
}

answered Dec 05 '13 at 10:00

Serve Laurijssen

9,266
5
45
98

I am skipping the first token intentionally (forgot to comment in code) as i don't want it (i will be using " as delimeter and it parses always first token before delimeter) ,i dont know if the programm internally ran as multi threaded, how do i correct it? – NxC Dec 05 '13 at 10:03

WhozCraig · Accepted Answer · 2013-12-05T10:59:06.927

Your supposition that token[0] = wcstok(buf, DELIMITER); will be NULL if there is no discovered delimiter is incorrect. There will be a token; the content of the string, and is easily proven in the following:

#include <iostream>
#include <cwchar>

int main()
{
    wchar_t data[] = L"NoDelimiter";
    wchar_t *ctx = nullptr, *token = wcstok(data, L"\"", &ctx);

    if (token != nullptr)
        std::wcout << token << std::endl;

    return 0;
}

Output

NoDelimiter

Therefore, this code:

token[0] = wcstok(buf, DELIMITER);
if (token[0]) // zero if line is blank

... is only correct if it is indeed a truly blank input (i.e. empty with NO characters). If there is even one character (such as a pesky newline), that eval will be true and you'll fall into your for-loop. But you can easily test if it is a truly blank line simply by doing this:

if (buff[0]) // true if line is not blank

Moving on, since you're purposely skipping the first token that brings us to:

for (n = 0; n < MAX_TOKENS_PER_LINE; n++)   
{
    token[n] = wcstok(0, DELIMITER); // will save NULL if original line had no delimiters
    if (!token[n]) 
        break;  // will break here since token[n] is NULL

In summary, your logic loop is flawed. Before you drop into skipping a token you had better be sure you actually read a token that was delimited with what you think was present. If nothing else I would advise you test to ensure the string has a '"' before assuming the reason the returned data was non-null is because one was present. As I showed above, that isn't the only way wcstok() will return a token.

Alternative Approach

There are dozens of options for doing what I think you're trying to do. This is just one of them, and you can adapt it as you see fit.

#include <iostream>
#include <sstream>
#include <algorithm>
#include <vector>

int main()
{
    wchar_t data[] = L"\"Test1\", \"Test2\", \"Test3\"";
    std::wistringstream iss(data);

    std::vector<std::wstring> tokens;
    std::wstring token;
    if (std::getline(iss, token, L'"') && std::getline(iss, token, L'"'))
    {
        while (iss)
        {
            // output the real token
            tokens.push_back(token);
            std::wcout << token << std::endl;
            std::getline(iss, token, L'"'); // skip separator to next "
            std::getline(iss, token, L'"'); // read through next "
        }
    }

    // tokens now has your "data" delimited tokens

    return 0;
}

Thanks @whozcraig , i will check this implementing in my code — NxC, Dec 05 '13 at 20:00

bad pointer: expression can not be evaluated while parsing tokens

2 Answers2