2

I'm currently trying to move some code away from using boost::variant in favour of std::variant, but have run into a problem that I can't figure out. Below is a minimal test case:

#include <string>
#include <variant>

#include <boost/spirit/home/x3.hpp>
#include <boost/variant/recursive_wrapper.hpp>
#include <boost/fusion/include/adapt_struct.hpp>

struct Recurse;
//using Base = boost::variant< // This works
using Base = std::variant<
    std::string,
    boost::recursive_wrapper<Recurse>>;

struct Recurse
{
    int _i;
    Base _base = std::string{};
};

BOOST_FUSION_ADAPT_STRUCT(
    Recurse,
    (int, _i),
    (Base, _base)
)

namespace x3 = boost::spirit::x3;
namespace ascii = boost::spirit::x3::ascii;

const x3::rule<class Base_, Base> base = "base";
const auto operand = *x3::char_("a-zA-Z0-9_") | base;
const auto base_def = (x3::int_ >> operand) | operand;

BOOST_SPIRIT_DEFINE(base)

int main()
{
    std::string text;
    Base result;
    x3::phrase_parse(std::begin(text), std::end(text), base, ascii::space, result);
    return 0;
}

Wandbox for the error

What I think is occurring is that the parser is trying to assign an int directly to a value of type Base, but since an int doesn't directly map to a std::string or a boost::recursive_wrapper<>, it gets upset (whereby upset I mean 11 pages of compiler errors). Somehow, boost::variant avoids this issue. Any clues please?

Tim Angus
  • 983
  • 11
  • 26

1 Answers1

3

Somehow boost::variant avoids the error.

Yeah. Boost variant has attribute propagation support.

Besides, boost::variant has special handling of boost::recursive_wrapper so it might be a double no-fly.

A good article about recursive std::variants is here https://vittorioromeo.info/index/blog/variants_lambdas_part_2.html

What's wrong with boost::variant?

If you want you can write some transformation traits, or even look into x3::variant - it might suit you better?

Live On Coliru

#include <string>
#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/support/ast/variant.hpp>
#include <boost/variant/recursive_wrapper.hpp>
#include <boost/fusion/include/adapt_struct.hpp>

namespace x3 = boost::spirit::x3;
namespace ascii = boost::spirit::x3::ascii;

struct Recurse;
using Base = x3::variant<
    std::string,
    x3::forward_ast<Recurse> >;

struct Recurse
{
    int _i;
    Base _base;
};

BOOST_FUSION_ADAPT_STRUCT(
    Recurse,
    (int, _i),
    (Base, _base)
)

const x3::rule<class Base_, Base> base = "base";
const auto operand = *x3::char_("a-zA-Z0-9_") | base;
const auto base_def = (x3::int_ >> operand) | operand;

BOOST_SPIRIT_DEFINE(base)

int main()
{
    std::string text;
    Base result;
    x3::phrase_parse(std::begin(text), std::end(text), base, ascii::space, result);
    return 0;
}

Side note: No x3::forward_ast<> does not help with std::variant, confirming that std::variant just lacks support in x3

UPDATE

You can work-around things by making your Base a derived struct with the required machinery to indicate to Spirit that it is a variant (and over which types). That way you don't have to go through trait specialization hell:

struct Recurse;

struct Base : std::variant<std::string, boost::recursive_wrapper<Recurse> > {
    using BaseV = std::variant<std::string, boost::recursive_wrapper<Recurse> >;
    using BaseV::BaseV;
    using BaseV::operator=;

    struct adapted_variant_tag {};
    using types = boost::mpl::list<std::string, Recurse>;
};

struct Recurse {
    int _i;
    Base _base;
};

As you can see, it's basically the same¹, but adds adapted_variant_tag and types nested types.

Note that by cleverly hardcoding the types sequence, we can pretend to handle the recursive wrapper smartly. We're lucky that this is enough to fool the system.

Adding some debug output and test-cases:

Live On Coliru

#include <string>
#include <variant>
#include <iostream>
#include <iomanip>
#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/support/ast/variant.hpp>
#include <boost/variant/recursive_wrapper.hpp>
#include <boost/fusion/include/adapt_struct.hpp>

namespace x3 = boost::spirit::x3;
namespace ascii = boost::spirit::x3::ascii;

namespace { // for debug
    template<class T>
    std::ostream& operator<<(std::ostream& os, boost::recursive_wrapper<T> const& rw) {
       return os << rw.get();
    }
    template<class... Ts>
    std::ostream& operator<<(std::ostream& os, std::variant<Ts...> const& sv) {
       std::visit([&os](const auto& v) { os << v; }, sv);
       return os;
    }
}

struct Recurse;

struct Base : std::variant<std::string, boost::recursive_wrapper<Recurse> > {
    using BaseV = std::variant<std::string, boost::recursive_wrapper<Recurse> >;
    using BaseV::BaseV;
    using BaseV::operator=;

    struct adapted_variant_tag {};
    using types = boost::mpl::list<std::string, Recurse>;
};

struct Recurse {
    int _i;
    Base _base;
    friend std::ostream& operator<<(std::ostream& os, Recurse const& r) {
        return os << "[" << r._i << ", " << r._base << "]";
    }
};

BOOST_FUSION_ADAPT_STRUCT(
    Recurse,
    (int, _i),
    (Base, _base)
)

static_assert(x3::traits::is_variant<Base>::value);
const x3::rule<class Base_, Base> base = "base";
const auto operand = *x3::char_("a-zA-Z0-9_") | base;
const auto base_def = (x3::int_ >> operand) | operand;

BOOST_SPIRIT_DEFINE(base)

int main()
{
    for (std::string const text : { "yeah8", "32 more" }) {
        Base result;
        auto f = begin(text), l = end(text);
        if (x3::phrase_parse(f, l, base, ascii::space, result)) {
            std::cout << "Result: " << result << "\n";
        } else {
            std::cout << "Failed\n";
        }

        if (f!=l) {
            std::cout << "Remaining input: " << std::quoted(std::string(f,l)) << "\n";
        }

    }
}

Which prints

Result: yeah8
Result: [32, more]

Update 2: Icing The Cake

Here's the traits required to make std::variant just work:

namespace boost::spirit::x3::traits {
    template<typename... t>
    struct is_variant<std::variant<t...> >
        : mpl::true_ {};

    template <typename attribute, typename... t>
    struct variant_has_substitute_impl<std::variant<t...>, attribute>
    {
        typedef std::variant<t...> variant_type;
        typedef typename mpl::transform<
              mpl::list<t...>
            , unwrap_recursive<mpl::_1>
            >::type types;
        typedef typename mpl::end<types>::type end;

        typedef typename mpl::find<types, attribute>::type iter_1;

        typedef typename
            mpl::eval_if<
                is_same<iter_1, end>,
                mpl::find_if<types, traits::is_substitute<mpl::_1, attribute>>,
                mpl::identity<iter_1>
            >::type
        iter;

        typedef mpl::not_<is_same<iter, end>> type;
    };


    template <typename attribute, typename... t>
    struct variant_find_substitute<std::variant<t...>, attribute>
    {
        typedef std::variant<t...> variant_type;
        typedef typename mpl::transform<
              mpl::list<t...>
            , unwrap_recursive<mpl::_1>
            >::type types;

        typedef typename mpl::end<types>::type end;

        typedef typename mpl::find<types, attribute>::type iter_1;

        typedef typename
            mpl::eval_if<
                is_same<iter_1, end>,
                mpl::find_if<types, traits::is_substitute<mpl::_1, attribute> >,
                mpl::identity<iter_1>
            >::type
        iter;

        typedef typename
            mpl::eval_if<
                is_same<iter, end>,
                mpl::identity<attribute>,
                mpl::deref<iter>
            >::type
        type;
    };

    template <typename... t>
    struct variant_find_substitute<std::variant<t...>, std::variant<t...> >
        : mpl::identity<std::variant<t...> > {};
}

That's a lot of noise but you can put it away in a header somewhere.

BONUS

Fixing the grammar:

  • you probably meant to have lexeme[] around the string production
  • you probably meant to have a minimal lenght of the string (+char_, not *char_) seeing that there are no delimiters
  • you may have have to reorder the branches because the string production would gobble up integers for recursed rules.

Here's my touched-up take on the grammar, where the rules closely mirror the AST, as usually makes sense:

namespace Parser {
    static_assert(x3::traits::is_variant<Base>::value);
    const x3::rule<class Base_, Base> base = "base";
    const auto string = x3::lexeme[+x3::char_("a-zA-Z0-9_")];
    const auto recurse = x3::int_ >> base;
    const auto base_def = recurse | string;
    BOOST_SPIRIT_DEFINE(base)
}

Simplify Fusion

Last but not least, in C++11 era you can deduce the adapted fusion members:

BOOST_FUSION_ADAPT_STRUCT(Recurse, _i, _base)

Live Full Demo

Live On Coliru

#include <string>
#include <variant>
#include <iostream>
#include <iomanip>
#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/support/ast/variant.hpp>
#include <boost/variant/recursive_wrapper.hpp>
#include <boost/fusion/include/adapt_struct.hpp>

namespace x3 = boost::spirit::x3;
namespace ascii = boost::spirit::x3::ascii;

namespace { // for debug
    template<class T>
    std::ostream& operator<<(std::ostream& os, boost::recursive_wrapper<T> const& rw) {
       return os << rw.get();
    }
    template<class... Ts>
    std::ostream& operator<<(std::ostream& os, std::variant<Ts...> const& sv) {
       std::visit([&os](const auto& v) { os << v; }, sv);
       return os;
    }
}

struct Recurse;
using Base = std::variant<
    std::string,
    boost::recursive_wrapper<Recurse> >;

namespace boost::spirit::x3::traits {
    template<typename... T>
    struct is_variant<std::variant<T...> >
        : mpl::true_ {};

    template <typename Attribute, typename... T>
    struct variant_has_substitute_impl<std::variant<T...>, Attribute>
    {
        typedef std::variant<T...> variant_type;
        typedef typename mpl::transform<
              mpl::list<T...>
            , unwrap_recursive<mpl::_1>
            >::type types;
        typedef typename mpl::end<types>::type end;

        typedef typename mpl::find<types, Attribute>::type iter_1;

        typedef typename
            mpl::eval_if<
                is_same<iter_1, end>,
                mpl::find_if<types, traits::is_substitute<mpl::_1, Attribute>>,
                mpl::identity<iter_1>
            >::type
        iter;

        typedef mpl::not_<is_same<iter, end>> type;
    };


    template <typename Attribute, typename... T>
    struct variant_find_substitute<std::variant<T...>, Attribute>
    {
        typedef std::variant<T...> variant_type;
        typedef typename mpl::transform<
              mpl::list<T...>
            , unwrap_recursive<mpl::_1>
            >::type types;

        typedef typename mpl::end<types>::type end;

        typedef typename mpl::find<types, Attribute>::type iter_1;

        typedef typename
            mpl::eval_if<
                is_same<iter_1, end>,
                mpl::find_if<types, traits::is_substitute<mpl::_1, Attribute> >,
                mpl::identity<iter_1>
            >::type
        iter;

        typedef typename
            mpl::eval_if<
                is_same<iter, end>,
                mpl::identity<Attribute>,
                mpl::deref<iter>
            >::type
        type;
    };

    template <typename... T>
    struct variant_find_substitute<std::variant<T...>, std::variant<T...> >
        : mpl::identity<std::variant<T...> > {};
}

static_assert(x3::traits::is_variant<Base>{}, "");

struct Recurse
{
    int _i;
    Base _base;
    friend std::ostream& operator<<(std::ostream& os, Recurse const& r) {
        return os << "[" << r._i << ", " << r._base << "]";
    }
};

BOOST_FUSION_ADAPT_STRUCT(Recurse, _i, _base)

namespace Parser {
    static_assert(x3::traits::is_variant<Base>::value);
    const x3::rule<class Base_, Base> base = "base";
    const auto string = x3::lexeme[+x3::char_("a-zA-Z0-9_")];
    const auto recurse = x3::int_ >> base;
    const auto base_def = recurse | string;
    BOOST_SPIRIT_DEFINE(base)
}

int main()
{
    for (std::string const text : { "yeah8", "32 more", "18 766 most" }) {
        Base result;
        auto f = begin(text), l = end(text);
        if (x3::phrase_parse(f, l, Parser::base, ascii::space, result)) {
            std::cout << "Result: " << result << "\n";
        } else {
            std::cout << "Failed\n";
        }

        if (f!=l) {
            std::cout << "Remaining input: " << std::quoted(std::string(f,l)) << "\n";
        }
    }
}

Which prints:

Result: yeah8
Result: [32, more]
Result: [18, [766, most]]

¹ (the subtle difference MAY bite you in generic programming where you need to access the base-class explicitly)

sehe
  • 374,641
  • 47
  • 450
  • 633
  • 3
    "What's wrong with boost::variant?" The main reason I'm trying to avoid it is actually because boost::apply_visitor is hurting compile times to an extent that std::visit apparently doesn't. I don't really have any problem with boost::variant besides that, though I guess in general it's better to avoid boost if a standard library feature can do the job. Thanks for your suggestions. – Tim Angus Apr 24 '20 at 17:01
  • Tough crowd. So, I'll update with a "hacky" kind of workaround that stops short of fullblown traits implementation (that is too heavy for my taste) – sehe Apr 24 '20 at 21:46
  • Update added ([live demo](http://coliru.stacked-crooked.com/a/dbe39b69a5a2350b)). – sehe Apr 24 '20 at 22:07
  • Aaaaand of course after struggling for 45 minutes and abandoning, I of course get the traits approach working as well: now every `std::variant` is supported with `recursive_wrapper` to boot: http://coliru.stacked-crooked.com/a/4c0f24486d3a952d. Also fixes a number of issues with the grammar. (Writing explanation in the answer) – sehe Apr 24 '20 at 22:17
  • Update 2: "Icing The Cake", and bonus ("Fixing the grammar" and "Simplify Fusion") added – sehe Apr 24 '20 at 22:24
  • Holy crap, you really went to town there. Thanks very much! I'm not sure the inheriting from `std::variant` approach will work for me since `std::visit` needs `std::variant_size` (amongst other things) to work on it's template parameter, which it won't if it's not strictly passed a `std::variant`. But the traits look to be spot on. Seems like they should live in spirit itself tbh? – Tim Angus Apr 27 '20 at 15:07
  • 1
    Ah, you actually pointed that out. "¹ (the subtle difference MAY bite you in generic programming where you need to access the base-class explicitly)" – Tim Angus Apr 27 '20 at 15:07
  • 1
    @TimAngus What I usually do in situations like that is to have an `auto& as_variant() { return static_cast(*this); }` member. That beats using magic. Yeah, I could send a PR to Spirit devs. I'm pretty sure they'd be interested in refactoring a bit before landing, because the whole thing was largely copy-paste. Also, the `recursive_wrapper` trick really seems out-of-place a bit for `std::variant` because it lives inside the Boost Variant library. Mmmm. – sehe Apr 27 '20 at 15:19
  • What's wrong with boost::variant? I would emphasis that this is a more general problem with spirit. It is closely related to other boost library like phenix. So if you want to have say a parsing library that parse a file and generate a structure representing the AST (and nothing more, the output of the lib is the AST object), then you have to do extra effort to transform the result of the spirit parsing to an 'de-boostified' object that will be the returned object of your API. – sandwood Nov 12 '20 at 13:12