How to tokenize string by delimiters?

Question

I need to tokenize string by delimiters.

For example:

For "One, Two Three,,, Four" I need to get {"One", "Two", "Three", "Four"}.

I am attempting to use this solultion https://stackoverflow.com/a/55680/1034253

std::vector<std::string> strToArray(const std::string &str,
                                    const std::string &delimiters = " ,")
{
    boost::char_separator<char> sep(delimiters.c_str());
    boost::tokenizer<boost::char_separator<char>> tokens(str.c_str(), sep);

    std::vector<std::string> result;
    for (const auto &token: tokens) {
        result.push_back(token);
    }

    return result;
}

But I get the error:

boost-1_57\boost/tokenizer.hpp(62): error C2228: left of '.begin' must have class/struct/union type is 'const char *const'

Does that error refer to one of the lines of code that you're showing us? — Drew Dormann, Feb 05 '15 at 22:25
The solution you linked to does not use `c_str()`. I'm assuming that boost requires that the argument is STL-centric, i.e. has a `begin()` iterator. — PaulMcKenzie, Feb 05 '15 at 22:26
@DrewDormann That error refer to tokenizer.hpp: template tokenizer(const Container& c,const TokenizerFunc& f) : first_(c.begin()), last_(c.end()), f_(f) { } — Ufx, Feb 05 '15 at 22:27

PaulMcKenzie · Accepted Answer · 2015-02-05T22:43:26.267

2

Change this:

boost::tokenizer<boost::char_separator<char>> tokens(str.c_str(), sep);

To this:

boost::tokenizer<boost::char_separator<char>> tokens(str, sep);

Link: http://www.boost.org/doc/libs/1_57_0/libs/tokenizer/tokenizer.htm

The container type requires a begin() function, and a const char* (which is what c_str()) returns does not meet this requirement.

edited Feb 05 '15 at 22:43

answered Feb 05 '15 at 22:30

PaulMcKenzie

34,698
4
24
45

I think boost::char_separator requires `char *`. – Ufx Feb 05 '15 at 22:42
@Ufx - ok. I removed the first line from the answer. – PaulMcKenzie Feb 05 '15 at 22:43

Drew Dormann · Answer 2 · 2015-02-05T22:37:44.180

1

Boost's tokenizer is probably overkill for the task you describe.

boost::split was written for this exact task.

std::vector<std::string> strToArray(const std::string &str,
                                    const std::string &delimiters = " ,")
{
    using namespace boost;
    std::vector<std::string> result;
    split( result, str, is_any_of(delimiters), token_compress_on );
    return result;
}

That optional token_compress_on signifies that your ,,, input shouldn't imply empty string tokens between those commas.

edited Feb 05 '15 at 22:37

answered Feb 05 '15 at 22:31

Drew Dormann

59,987
13
123
180

I've seen this solution. Can I stop adding tokens, when some size is reached? – Ufx Feb 05 '15 at 22:39
@Ufx It's done in one shot, so if you want a maximum size for the returned vector, you would resize it before returning. – Drew Dormann Feb 05 '15 at 22:44

sanfranciscan · Answer 3 · 2015-09-13T05:06:35.283

1

Short way.

string tmp = "One, Two, Tree, Four";
int pos = 0;
while (pos = tmp.find(", ") and pos > 0){
    string s = tmp.substr(0, pos);
    tmp = tmp.substr(pos+2);
    cout << s;
}

edited Sep 13 '15 at 05:06

answered Sep 13 '15 at 04:58

sanfranciscan

11
2

score 0 · Answer 4 · edited May 23 '17 at 11:44

I see a lot of boost answers, so I thought I'd supply a non-boost answer:

template <typename OutputIter>
void Str2Arr( const std::string &str, const std::string &delim, int start, bool ignoreEmpty, OutputIter iter )
{
    int pos = str.find_first_of( delim, start );
    if (pos != std::string::npos) {
        std::string nStr = str.substr( start, pos - start );
        trim( nStr );

        if (!nStr.empty() || !ignoreEmpty)
            *iter++ = nStr;
        Str2Arr( str, delim, pos + 1, ignoreEmpty, iter );
    }
    else
    {
        std::string nStr = str.substr( start, str.length() - start );
        trim( nStr );

        if (!nStr.empty() || !ignoreEmpty)
          *iter++ = nStr;
    }
}

std::vector<std::string> Str2Arr( const std::string &str, const std::string &delim )
{
    std::vector<std::string> result;
    Str2Arr( str, delim, 0, true, std::back_inserter( result ) );
    return std::move( result );
}

trim can be any trim function, I used this SO answer. It makes use of std::back_inserter and recursion. You could easily do it in a loop, but this sounded like more fun :)

How to tokenize string by delimiters?

4 Answers4