Boost::tokenizer comma separated (c++)

Question

Should be an easy one for you guys.....

I'm playing around with tokenizers using Boost and I want create a token that is comma separated. here is my code:

    string s = "this is, , ,  a test";
boost::char_delimiters_separator<char> sep(",");
boost::tokenizer<boost::char_delimiters_separator<char>>tok(s, sep);


for(boost::tokenizer<>::iterator beg= tok.begin(); beg!=tok.end(); ++beg)
{
    cout << *beg << "\n";
}

The output that I want is:

This is


 a test

What I am getting is:

This
is
,
,
,
a
test

UPDATED

Similar to this question: If I have `this is,,,a test` and `cout << "<" << *beg << "> ";` modified in your code, how do I get empty strings too, as ` <> <> `? — thedarkside ofthemoon, Mar 11 '14 at 16:52

score 16 · Accepted Answer · edited Feb 11 '19 at 20:28

16

You must give the separator to tokenizer!

boost::tokenizer<boost::char_delimiters_separator<char>>tok(s, sep);

Also, replace the deprecated char_delimiters_separator with char_separator:

string s = "this is, , ,  a test";
boost::char_separator<char> sep(",");
boost::tokenizer< boost::char_separator<char> > tok(s, sep);
for(boost::tokenizer< boost::char_separator<char> >::iterator beg = tok.begin(); beg != tok.end(); ++beg)
{
    cout << *beg << "\n";
}

Please note that there is also a template parameter mismatch: it's good habit to typedef such complex types: so the final version could be:

string s = "this is, , ,  a test";
boost::char_separator<char> sep(",");
typedef boost::tokenizer< boost::char_separator<char> > t_tokenizer;
t_tokenizer tok(s, sep);
for (t_tokenizer::iterator beg = tok.begin(); beg != tok.end(); ++beg)
{
    cout << *beg << "\n";
}

edited Feb 11 '19 at 20:28

relt

103
3

answered Oct 29 '11 at 21:12

CapelliC

59,646
5
47
90

also, Now my output separates by comma AND spaces,I want to ignore spaces and I don't want to include the comma in the output (see updated question above)... – Lexicon Oct 29 '11 at 21:18
2

instead of `typedef`ing things just to use iterators you can use `auto` – Daniel Oct 31 '11 at 01:30
@Dani, actually you can't. `auto` is valid only in c++11 standard – kaspersky Dec 10 '13 at 12:12

score 0 · Answer 2 · answered Aug 25 '23 at 14:32

With modern C++, a much simpler formulation is possible than the other answer provides:

std::string my_string = "this is, , ,  a test";

for(auto const &substring : boost::tokenizer{my_string, boost::char_separator{","}}) {
  std::cout << substring << std::endl;
}

Boost::tokenizer comma separated (c++)

2 Answers2

Linked