How to push all the arguments into result vector when parsing with Spirit::Qi?

Question

I have several commands in a script language, so I need to parse them. During parsing, I would like to check that syntax is correct and the type of commands and its arguments (there is a variable number of arguments per script command type, so I use a std::vector<std::string> to store them).

I have had problems because when parsing, Only the first string is included into the vector, whatever the real numbers of strings exists.

Also, I have had to use a qi::as_string rule in all the arguments in order compiler works.

A minimal working example of my project is shown next:

//#define BOOST_SPIRIT_DEBUG
#include <boost/algorithm/string.hpp>
#include <boost/algorithm/string/predicate.hpp>
#include <boost/spirit/include/qi.hpp>
#include <iomanip>
#include <iostream>
#include <sstream>

namespace qi = boost::spirit::qi;

enum class TYPE {
    NONE,
    CMD1,
    CMD2,
    FAIL
};

struct Command {
    TYPE type = TYPE::NONE;
    std::vector<std::string> args;
};

using Commands = std::vector<Command>;

BOOST_FUSION_ADAPT_STRUCT(Command, type, args)

template <typename It>
class Parser : public qi::grammar<It, Commands()>
{
  private:
    qi::rule<It, Command(), qi::blank_type> none, cmd1, cmd2, fail;
    qi::rule<It, Commands()> start;

  public:
    Parser() : Parser::base_type(start)
    {
        using namespace qi;

        none = omit[*blank] >> &(eol | eoi)
            >> attr(TYPE::NONE)
            >> attr(std::vector<std::string>{});

        cmd1 = lit("CMD1") >> '('
            >> attr(TYPE::CMD1)
            >> as_string[lexeme[+~char_(")\r\n")]] >> ')';

        cmd2 = lit("CMD2") >> '('
            >> attr(TYPE::CMD2)
            >> as_string[lexeme[+~char_(",)\r\n")]] >> ','
            >> as_string[raw[double_]] >> ')';


        fail = omit[*~char_("\r\n")] //
            >> attr(TYPE::FAIL);

        start = skip(blank)[(none | cmd1 | cmd2 | fail) % eol] > eoi;
    }
};

Commands parse(std::string text)
{
    std::istringstream in(std::move(text));
    using It = boost::spirit::istream_iterator;

    static const Parser<It> parser;

    Commands commands;
    It first(in >> std::noskipws), last;//No white space skipping

    if (!qi::parse(first, last, parser, commands))
        // throw std::runtime_error("command parse error")
        ;

    return commands;
}

int main()
{
    std::string test{
R"(CMD1(some ad hoc text)
CMD2(identity, 25.5))"};

    try {
        auto commands = parse(test);
        std::cout << "elements: " << commands.size() << std::endl;
        std::cout << "CMD1 args: " << commands[0].args.size() << std::endl;
        std::cout << "CMD2 args: " << commands[1].args.size() << std::endl;// Error! Should be 2!!!!!

    } catch (std::exception const& e) {
        std::cout << e.what() << "\n";
    }
}

Also, here is a link to compiler explorer: https://godbolt.org/z/qM6KTcTTK

Any help fixing this? Thanks in advance

sehe · Accepted Answer · 2021-10-26T20:48:28.220

Enabling your debugging shows: https://godbolt.org/z/o3nvjz9bG

Not clear enough for me. Let's add an argument rule:

struct Command {
    using Arg  = std::string;
    using Args = std::vector<Arg>;
    enum TYPE { NONE, CMD1, CMD2, FAIL };

    TYPE type = NONE;
    Args args;
};

qi::rule<It, Command::Arg()> arg;

And

none = omit[*blank] >> &(eol | eoi)
    >> attr(Command::NONE)
    /*>> attr(Command::Args{})*/;

arg  = raw[double_] | +~char_(",)\r\n");

cmd1 = lit("CMD1") >> attr(Command::CMD1) //
    >> '(' >> arg >> ')';

cmd2 = lit("CMD2") >> attr(Command::CMD2) //
    >> '(' >> arg >> ',' >> arg >> ')';

fail = omit[*~char_("\r\n")] //
    >> attr(Command::FAIL);

Now we can see https://godbolt.org/z/3Kqr3K41v

  <cmd2>
    <try>CMD2(identity, 25.5)</try>
    <arg>
      <try>identity, 25.5)</try>
      <success>, 25.5)</success>
      <attributes>[[i, d, e, n, t, i, t, y]]</attributes>
    </arg>
    <arg>
      <try>25.5)</try>
      <success>)</success>
      <attributes>[[2, 5, ., 5]]</attributes>
    </arg>
    <success></success>
    <attributes>[[CMD2, [[i, d, e, n, t, i, t, y]]]]</attributes>
  </cmd2>

Clearly, both arguments are parsed, but only one is assigned. The sad fact is that you're actively confusing the rule, by adapting a two-element struct and parsing a sequence of 3 elements.

You can get this to work, but you'd have help it (e.g. with transform_attribute, attr_cast<> or a separate rule):

    arg  = raw[double_] | +~char_(",)\r\n");
    args = arg % ',';

    cmd1 = lit("CMD1") >> attr(Command::CMD1) //
        >> '(' >> arg >> ')';

    cmd2 = lit("CMD2") >> attr(Command::CMD2) //
        >> '(' >> args >> ')';

Now you get:

  <cmd2>
    <try>CMD2(identity, 25.5)</try>
    <args>
      <try>identity, 25.5)</try>
      <arg>
        <try>identity, 25.5)</try>
        <success>, 25.5)</success>
        <attributes>[[i, d, e, n, t, i, t, y]]</attributes>
      </arg>
      <arg>
        <try> 25.5)</try>
        <success>)</success>
        <attributes>[[ , 2, 5, ., 5]]</attributes>
      </arg>
      <success>)</success>
      <attributes>[[[i, d, e, n, t, i, t, y], [ , 2, 5, ., 5]]]</attributes>
    </args>
    <success></success>
    <attributes>[[CMD2, [[i, d, e, n, t, i, t, y], [ , 2, 5, ., 5]]]]</attributes>
  </cmd2>

Now this hints at an obvious improvement: improve the grammar by simplifying:

    none  = omit[*blank] >> &(eol | eoi) >> attr(Command{Command::NONE, {}});
    fail  = omit[*~char_("\r\n")] >> attr(Command::FAIL);

arg   = raw[double_] | +~char_(",)\r\n");
args  = '(' >> arg % ',' >> ')';
cmd   = no_case[type_] >> -args;

start = skip(blank)[(cmd|fail) % eol] > eoi;

Then add validation to the commands after the fact.

Demo

Live On Compiler Explorer

//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <iomanip>
#include <iostream>

namespace qi = boost::spirit::qi;

struct Command {
    using Arg  = std::string;
    using Args = std::vector<Arg>;
    enum Type { NONE, CMD1, CMD2, FAIL };

    Type type = NONE;
    Args args;

    friend std::ostream& operator<<(std::ostream& os, Type type) {
        switch(type) {
            case NONE: return os << "NONE";
            case CMD1: return os << "CMD1";
            case CMD2: return os << "CMD2";
            case FAIL: return os << "FAIL";
            default: return os << "???";
        }
    }
    friend std::ostream& operator<<(std::ostream& os, Command const& cmd) {
        os << cmd.type << "(";
        auto sep = "";
        for (auto& arg : cmd.args)
            os << std::exchange(sep, ", ") << std::quoted(arg);
        return os << ")";
    }
};
using Commands = std::vector<Command>;

BOOST_FUSION_ADAPT_STRUCT(Command, type, args)

template <typename It> struct Parser : qi::grammar<It, Commands()> {
    Parser() : Parser::base_type(start) {
        using namespace qi;

        none  = omit[*blank] >> &(eol | eoi) >> attr(Command{Command::NONE, {}});
        fail  = omit[*~char_("\r\n")] >> attr(Command::FAIL);

        arg   = raw[double_] | +~char_(",)\r\n");
        args  = '(' >> arg % ',' >> ')';
        cmd   = no_case[type] >> -args;

        start = skip(blank)[(cmd|none|fail) % eol] > eoi;

        BOOST_SPIRIT_DEBUG_NODES((start)(fail)(none)(cmd)(arg)(args))
    }

private:
    struct type_sym : qi::symbols<char, Command::Type> {
        type_sym() { this->add//
            ("cmd1", Command::CMD1)
            ("cmd2", Command::CMD2);
        }
    } type;
    qi::rule<It, Command::Arg()>            arg;
    qi::rule<It, Command::Args()>           args;
    qi::rule<It, Command(), qi::blank_type> cmd, none, fail;
    qi::rule<It, Commands()>                start;
};

Commands parse(std::string const& text)
{
    using It = std::string::const_iterator;
    static const Parser<It> parser;

    Commands commands;
    It first = text.begin(), last = text.end();

    if (!qi::parse(first, last, parser, commands))
        throw std::runtime_error("command parse error");

    return commands;
}

int main()
{
    try {
        for (auto& cmd : parse(R"(
CMD1(some ad hoc text)
this is a bogus line
cmd2(identity, 25.5))"))
            std::cout << cmd << "\n";
    } catch (std::exception const& e) {
        std::cout << e.what() << "\n";
    }
}

Prints

NONE()
CMD1("some ad hoc text")
FAIL()
CMD2("identity", " 25.5")

If you insist on strong typing arguments, why not strongtyype the commands and use a rule per command? See e.g. https://stackoverflow.com/a/44261883/85371 — sehe, Oct 26 '21 at 16:27
Thank you for your advises!!! By the way, when I used boost::spirit::classic there was a push_back semantic action to use for pushing the matches inside the vector. I wonder if that method should be applicable for this case... — Pablo, Oct 26 '21 at 19:14
I also have found a terrible hack for doing this while reviewing the available qi parsers. Using repeat(1)[A] does the trick, due it returns a vector, but it's something I would not be proud to use in production code. — Pablo, Oct 26 '21 at 20:11
Correct. You could have used semantic actions. However, in my humble opinion the need for semantic actions usually ends up refuting the purpose of the library. I would not feel too bad about `repeat[p]` in some context, as it is about as ugly as `attr_cast`/`transform_attribute` or a separate rule. However my criterion would be for the code to _express_ the intent, which I think favors the subrule approach here. — sehe, Oct 26 '21 at 20:48

How to push all the arguments into result vector when parsing with Spirit::Qi?

1 Answers1

Demo