Using Boost Tokenizer escaped_list_separator with different parameters

Question

Hello i been trying to get a tokenizer to work using the boost library tokenizer class. I found this tutorial on the boost documentation:

http://www.boost.org/doc/libs/1 _36 _0/libs/tokenizer/escaped _list _separator.htm

problem is i cant get the argument's to escaped _list _separator("","","");

but if i modify the boost/tokenizer.hpp file it work's. but that's not and ideal solution was wondering if there's anything i am missing to get diferent arguments into the escaped _list _separator.

i want to make it split on spaces with " and ' for escaping and with no escape character inside the quoted string.

this is used for a argument parsing system in a ingame console system.

#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>

int main()
{
    using namespace std;
    using namespace boost;
    string s = "exec script1 \"script argument number one\"";
    string separator1("");//dont let quoted arguments escape themselves
    string separator2(" ");//split on spaces
    string separator3("\"\'");//let it have quoted arguments
    tokenizer<escaped_list_separator<char>(separator1,separator2,separator3)> tok(s);
    for(tokenizer<escaped_list_separator<char>(separator1,separator2,separator3)>::iterator beg=tok.begin(); beg!=tok.end();++beg)
    {
        cout << *beg << "\n";
    }
}

the error from visual studio 2005 is error C2974: 'boost::tokenizer' : invalid template argument for 'TokenizerFunc', type expected

EDIT: This question was awnsered by ferrucio and explained by peter thank's everybody.

Rather than editing to indicate which answer you like, please make it as your accepted answer (using the check-mark button on the left of the answer you like). — John Zwinck, Feb 25 '09 at 01:44

score 15 · Accepted Answer · answered Feb 12 '09 at 15:42

15

try this:

#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>

int main()
{
    using namespace std;
    using namespace boost;
    string s = "exec script1 \"script argument number one\"";
    string separator1("");//dont let quoted arguments escape themselves
    string separator2(" ");//split on spaces
    string separator3("\"\'");//let it have quoted arguments

    escaped_list_separator<char> els(separator1,separator2,separator3);
    tokenizer<escaped_list_separator<char>> tok(s, els);

    for(tokenizer<escaped_list_separator<char>>::iterator beg=tok.begin(); beg!=tok.end();++beg)
    {
        cout << *beg << "\n";
    }
}

answered Feb 12 '09 at 15:42

Ferruccio

98,941
38
226
299

Thank's this actually worked i knew it was something small :P. – Annerajb Feb 12 '09 at 15:52
2

it is possible to keep `"` in output, for example `exec | script1 | "script argument number one"` ? – triclosan May 28 '12 at 08:11
@Ferruccio is there a good reason for not having `separator1("\\")`? That would allow one to escape a quote character inside a string. – mgd Aug 27 '13 at 09:40
@mgd - No, there isn't. My answer simply addressed the cause of the compilation error (the misplaced `>`). I left everything else as is. – Ferruccio Aug 27 '13 at 12:27

pk. · Answer 2 · 2009-02-12T19:28:51.977

5

It seems like you're declaring your tokenizer type incorrectly.

typedef boost::tokenizer< boost::escaped_list_separator<char> > Tokenizer;
boost::escaped_list_separator<char> Separator( '\\', ' ', '\"' );
Tokenizer tok( s, Separator );

for( Tokenizer::iterator iter = tok.begin(); iter != tok.end(); ++iter )
{ cout << *iter << "\n"; }

You want to make a boost::tokenizer< boost::escaped_list_separator< char > > typed object with a boost::escaped_list_separator< char > separator object as its TokenizerFunc.

edited Feb 12 '09 at 19:28

answered Feb 12 '09 at 15:33

pk.

420
2
6

Those should be single quotes in your constructor - the double quotes are for char*, not char. – Harper Shelby Feb 12 '09 at 15:37

score 0 · Answer 3 · answered Jan 24 '17 at 13:37

A relevant point yet not the answer for this is that if user want to output a double quote in the result, the embedded quote ("") described in the Wikipedia (like here) should be replaced with a string of (\\\"), where \\ means a escape char and \" means the quote mark. In this way, the quote mark will be displayed in the output result. The following code snippet is an example.

typedef boost::escaped_list_separator<char> std_token;
typedef boost::tokenizer<boost::escaped_list_separator<char>> tokenizer;

std_token a_token( "\\", ",", "\"" );
std::string s = "\"Boost,\\\" C++ Libraries\" ";
tokenizer tok{ s, a_token };
for ( const auto &t : tok )
    std::cout << t << '\n';

This is a typical output for this

Using Boost Tokenizer escaped_list_separator with different parameters

3 Answers3

Linked