2

When i try to compile the code below (using the example from boost\spirit\home\lex\argument.hpp: value_setter) I get the following compiler error:

c:\program files (x86)\boost\boost_1_50\boost\range\iterator.hpp(63) : error C2039: 'type' : is not a member of 'boost::mpl::eval_if_c<C,F1,F2>'
with
[
    C=true,
    F1=boost::range_const_iterator<const char *>,
    F2=boost::range_mutable_iterator<const char *const >
]
c:\program files (x86)\boost\boost_1_50\boost\range\iterator_range_core.hpp(56) : see reference to class template instantiation 'boost::range_iterator<C>' being compiled
with
[
    C=const char *const 
]
...

Without the semantic action, everything compiles fine. Here is the example:

#include <boost/spirit/include/lex_lexertl.hpp>
namespace lex = boost::spirit::lex;

template <typename Lexer>
struct my_tokens : lex::lexer<Lexer>
{
    my_tokens()
    {
        identifier = "[a-zA-Z_][a-zA-Z0-9_]*";
        this->self = identifier  [ lex::_val = "identifier" ] // problematic action
                   ;
    }
    lex::token_def<> identifier;
};

int main()
{
    typedef std::string::iterator base_iterator_type;
    typedef
        lex::lexertl::actor_lexer<lex::lexertl::token<base_iterator_type> > 
        lexer_type;

    my_tokens<lexer_type> myLexer;
    std::string str = "id1";
    base_iterator_type first = str.begin();
    bool r = lex::tokenize(first, str.end(), myLexer);

    if (!r) {
        std::string rest(first, str.end());
        std::cerr << "Lexical analysis failed\n" << "stopped at: \"" 
                  << rest << "\"\n";
    }
}

What is going wrong? How can I set/change the value of a token?

coproc
  • 6,027
  • 2
  • 20
  • 31

1 Answers1

3

Your token_def should expose the expected attribute type (the compile error suggests you are assigning a string literal to an iterator_range):

lex::token_def<std::string> identifier;

Now, match up the type in the assignment

this->self = identifier  [ lex::_val = std::string("identifier") ]

Don't forget to update the token type to reflect the set of possible token attribute types:

typedef
    lex::lexertl::actor_lexer<lex::lexertl::token<base_iterator_type,
        boost::mpl::vector<std::string> > > 
    lexer_type;

Now it should compile:

#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/phoenix.hpp>
namespace lex = boost::spirit::lex;
namespace phx = boost::phoenix;

template <typename Lexer>
struct my_tokens : lex::lexer<Lexer>
{
    my_tokens()
    {
        identifier = "[a-zA-Z_][a-zA-Z0-9_]*";
        this->self = identifier  [ lex::_val = std::string("identifier") ]
                   ;
    }
    lex::token_def<std::string> identifier;
};

int main()
{
    typedef std::string::iterator base_iterator_type;
    typedef
        lex::lexertl::actor_lexer<lex::lexertl::token<base_iterator_type, boost::mpl::vector<std::string> > > 
        lexer_type;

    my_tokens<lexer_type> myLexer;
    std::string str = "id1";
    base_iterator_type first = str.begin();
    bool r = lex::tokenize(first, str.end(), myLexer);

    if (!r) {
        std::string rest(first, str.end());
        std::cerr << "Lexical analysis failed\n" << "stopped at: \"" 
                  << rest << "\"\n";
    }
}
sehe
  • 374,641
  • 47
  • 450
  • 633
  • The possibility of adding attribute types to the token type definition is tricky. The documentation on this subject is rather short. That was the key point I was missing. Thanks for pointing out! – coproc May 14 '13 at 08:51
  • Is it somehow possible to form the token value based on matching groups in the token regex? Or at least to get access to them? – Adam Badura Feb 18 '16 at 08:05
  • I ended making a separate question for this: http://stackoverflow.com/questions/35476454/how-to-make-boost-spirit-lex-token-value-be-a-substring-of-matched-sequence-pre – Adam Badura Feb 18 '16 at 08:37