Spirit: discarding attribute during backtracking

Question

I don't understand the behaviour of x3 in the following example (taken from a larger grammar).

The grammar is a bit weird, granted, but roughly it implements (lal)?(<char>)?. When the second group is not present, it defaults to <default>. I don't understand why on the input "lal<char>" I get defaultchar as a result:

#include <iostream>

#include <boost/config/warning_disable.hpp>
#include <boost/spirit/home/x3.hpp>
#include <boost/fusion/adapted/std_pair.hpp>

int main()
{
  namespace x3 = boost::spirit::x3;
  namespace ascii = boost::spirit::x3::ascii;

  using ascii::alnum;
  using ascii::char_;
  using ascii::space;
  using x3::attr;
  using x3::eoi;
  using x3::lexeme;
  using x3::lit;
  using x3::string;

  const auto letter
    = lit("<") >> string("char") >> lit('>')
    | attr(std::string{"default"})
    ;

  const auto letterset
    = letter >> eoi
    | lit("lal") >> letter >> eoi
    ;

  for (std::string i: {"", "<char>", "lal", "lal<char>"})
    {
      auto res = std::string{};
      auto first = i.cbegin();
      auto last = i.cend();
      auto r = x3::phrase_parse(first, last, letterset, space, res);
      if (r && first == last)
        std::cout << i << ": " << res << '\n';
      else
        std::cout << i << ": failed\n";
    }
}

result:

: default
<char>: char
lal: defaultdefault
lal<char>: defaultchar

I do notice that if I swap the alternatives of letterset, then I get the expected result, which is probably a sign that's related to backtracking: in the original case I get the default from its first attempt, and the char from the second one, successful.

I'm very tempted to call this a bug, but I'm an x3 newbie...

This is X3 from Boost 1.65.

Thanks in advance @sehe :)

It's seems that https://stackoverflow.com/questions/39537675/strange-behaviour-of-parser-with-boost-spirit-x3 has hints on what is going on. — akim, Jan 21 '18 at 12:20

score 3 · Accepted Answer · answered Mar 15 '18 at 21:17

Yes, you've hit a side effect of backtracking. Spirit parsers append their results if attribute is a container (std::string is a container), but if a branch fails the container is not cleared.

There is hold directive in Qi to deal with it, but not in X3. I will look what I can do to X3, I think there are some solutions with close to zero overhead without the horrible hold directive.

The general suggestion is to try to rewrite you grammar to not backtrack, but if it is impossible you may use a workaround (it should be a slightly more performant alternative to hold):

const auto recover_from_backtrack = (rule<class _, std::string>{} = eps) /
    [](auto const& ctx) { x3::_attr(ctx).clear(); x3::_pass(ctx) = false; };

const auto letterset
    = letter >> eoi
    | recover_from_backtrack
    | lit("lal") >> letter >> eoi
    ;

So is it considered "ok" that backtracking leaves side-effects in x3's own generated attributes? As a simple user, I definitely see that as a bug^H^H^Hmisfeature. — akim, Mar 16 '18 at 09:53
I agree with you and I am going to change it. More info here https://github.com/boostorg/spirit/issues/378 — Nikita Kniazev, Mar 16 '18 at 19:19

Spirit: discarding attribute during backtracking

1 Answers1