Map input to ast types in boost spirit

Question

I want to implement color highlighting of input string that is being fed to given spirit grammar. Is there an easy (or any, if not easy) way to map given character from input into rule/ast type that it matches? Preferable in form of array/vector of rule/ast types where index is index of character of input string. Or maybe better - iterator rages to ast types.

score 1 · Accepted Answer · answered Jul 09 '18 at 22:18

Of course there is. Several answers on this site demonstrate similar things. You'll have to decide how you want to treat subrules.

A random example using on_success:

Live On Coliru

//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <iomanip>
namespace qi = boost::spirit::qi;

using It     = std::string::const_iterator;
using R      = boost::iterator_range<It>;
using RuleId = void const*;

struct token {
    R what;
    RuleId r_id;
}; 

struct assocociate_f {
    std::vector<token>& into;
    RuleId r_id = nullptr;

    template <typename Ctx>
    void operator()(qi::unused_type, Ctx& ctx) const {
        using boost::fusion::at_c;
        into.push_back({at_c<0>(ctx.attributes), r_id});
    }
};

int main() {
    qi::rule<It, R()> numlit, ident, oper;
    qi::rule<It, R(), qi::space_type> simple, expr;

    numlit = qi::raw[qi::double_];
    ident  = qi::raw[qi::lexeme[qi::alpha >> *qi::alnum]];
    simple = qi::raw[(numlit | ident | '(' >> expr >> ')')];
    oper   = qi::raw[qi::char_("-+*/%")];
    expr   = qi::raw[simple >> *(oper >> expr)];

    std::vector<token> collect;
    qi::on_success(numlit, assocociate_f{collect, &numlit});
    qi::on_success(ident,  assocociate_f{collect, &ident});
    qi::on_success(oper,   assocociate_f{collect, &oper});
    //qi::on_success(simple, assocociate_f{collect, &simple});
    //qi::on_success(expr,   assocociate_f{collect, &expr});

    BOOST_SPIRIT_DEBUG_NODES((numlit)(ident)(simple)(expr));

    auto idof = [&](token const& tok) -> std::string {
        auto match = [&](auto const& x) { return tok.r_id == static_cast<void const*>(&x); };
        if (match(numlit)) return "numeric literal";
        if (match(ident))  return "identifier";
        if (match(simple)) return "simple expression";
        if (match(expr))   return "expression";
        if (match(oper))   return "operator";
        return "other";
    };

    for (std::string const input : { "3 * pi + (13/47 - 5)" }) {
        std::cout << std::setw(20) << "input: " << input << "\n";
        It f = input.begin(), l = input.end();

        if (qi::phrase_parse(f, l, expr, qi::space)) {
            for (auto& tok : collect) {
                std::cout 
                    << std::setw(20) << idof(tok) + ": "
                    << std::setw(tok.what.begin() - input.begin() + 1) << tok.what
                    << "\n";
            }
        } else {
            std::cout << "Parse failed\n";
        }

        if (f!=l) {
            std::cout << "Remaining: '" << std::string(f,l) << "'\n";
        }
    }
}

Prints

             input: 3 * pi + (13/47 - 5)
   numeric literal: 3
          operator:   *
        identifier:     pi
          operator:        +
   numeric literal:           13
          operator:             /
   numeric literal:              47
          operator:                 -
   numeric literal:                   5

Uncommenting the extra

//qi::on_success(simple, assocociate_f{collect, &simple});
//qi::on_success(expr,   assocociate_f{collect, &expr});

You get: Live On Coliru

             input: 3 * pi + (13/47 - 5)
   numeric literal: 3
 simple expression: 3
          operator:   *
        identifier:     pi
 simple expression:     pi
          operator:        +
   numeric literal:           13
 simple expression:           13
          operator:             /
   numeric literal:              47
 simple expression:              47
          operator:                 -
   numeric literal:                   5
 simple expression:                   5
        expression:                   5
        expression:              47 - 5
        expression:           13/47 - 5
 simple expression:          (13/47 - 5)
        expression:          (13/47 - 5)
        expression:     pi + (13/47 - 5)
        expression: 3 * pi + (13/47 - 5)

More

A more funky example would be How to provider user with autocomplete suggestions for given boost::spirit grammar? - where string_view or string_ref is being used instead of iterator_range. Also, that "collapses" adjacent ranges to result in more usable ranges.

Other related examples:

This actually stores information inside AST during parsing... I have already enrolled AST, I'd prefer not to change it — PiotrK, Jul 10 '18 at 06:31
If you share a minimal code sample, it becomes easier to be helpful with the answer code. — sehe, Jul 10 '18 at 09:52
I meant the AST generated from above parser is useless - it does not contain any real parsed data (like numbers typed as doubles), only reference to source parser. I was trying to store both result (address and parsed tree), but it's actually pretty tought thing to do it — PiotrK, Jul 10 '18 at 16:00
I've done it on occasion, probably even on this site. Again, if you get stuck with that, please share a minimal example. — sehe, Jul 10 '18 at 16:01
Big thanks for that link, that was exactly what I was looking for! :-) — PiotrK, Jul 11 '18 at 13:12
I'm pretty sure there's more examples (but SO search is... pretty poor). Again, if you run into a block, feel free to come back with some more pointed questions. — sehe, Jul 11 '18 at 13:14

Map input to ast types in boost spirit

1 Answers1

More

Linked