How to provide user with advanced autocomplete suggestions for given boost::spirit grammar?

Question

This actually builds up after my previous question:

How to provider user with autocomplete suggestions for given boost::spirit grammar?

To Sehe solution I've added annotater for syntax coloring:

First by adopting hints structure to this (note: ast_type_t is an enum class):

template <class ast_type_t> class hints_t
{
    struct by_location_t
    {
        template<typename T, typename U> bool operator()(T const& a, U const& b) const
        {
            if(loc(a) == loc(b)) return size(a) > size(b);
            return loc(a) < loc(b);
        }

    private:
        static location_t loc(source_t const& s)
        {
            return s.begin();
        }
        static location_t loc(location_t const& l)
        {
            return l;
        }
        static std::size_t size(source_t const& s)
        {
            return s.end() - s.begin();
        }
        static std::size_t size(location_t const& s)
        {
            return 1;
        }
    };

public:
    std::map<location_t, std::string, by_location_t> incomplete;
    std::map<source_t, candidates_t, by_location_t>  suggestions;
    std::map<source_t, ast_type_t, by_location_t>    annotations;

    operator bool() const
    {
        return incomplete.size() || suggestions.size();
    }
};

Then by adding annotater itself:

struct annotation_t
{
    typedef void              result_type;
    hints_t* hints;

    template<typename first_t, typename last_t>
    void operator()(ast::type_t id, first_t f, last_t l) const
    {
        if (hints)
        {
            source_t loc(&*f, l - f);
            if (loc.size() > 0)
            {
                auto inserted = hints->annotations.emplace(loc, id);
                if (!inserted.second) inserted.first->second = id;
            }
        }
    }
};
::boost::phoenix::function<annotation_t> annotate{annotation_t{hints}};

And lastly by including them in parser itself:

on_success(lang_text, annotate(ast::type_t::lang_text, ::boost::spirit::qi::_1, ::boost::spirit::qi::_3));

This generally works, but I've hit another road blocker; I need autocompletion that actually can build dynamic list of suggestion based on previously parsed values; The best example would be to use foo.member1.member2 from sehe comment on original question. I do not allow member definition inside parsed text itself, they are always provided by external C++ class;

An example of such C++ class:

class member_provider_t
{
     public: virtual ~member_provider_t() {}
     public: virtual std::vector<std::string> possible_identifiers_after(...) = 0;
}

Now it get tricky - I would like to pass entire parse result (up to this point) into to ... placeholder (in form of parsed ast). Example: parsing uri of custom schema:

schema://main.menu.screen/show/ab... (suggest: about*)

(*about - This depends on entire url parsed so far)

If this is not possible, then any other way to pass as many parsed parts as possible to provider function would suit me.

Answer 1

In relation to syntax highlighting, have you seen these?

• Map input to ast types in boost spirit
• Cannot get Boost Spirit grammar to use known keys for std::map<>

In relation to the second part of the question about context-sensitive completion, you simply need semantic analysis. The most important thing to remember here is to separate parsing and semantic analysis to simplify your life.

In the original answer we already have an AST capable of representing partially correct inputs. This is all you need.

Conceptually, I'd try to map the cursor position back to the corresponding AST node, and pass that to the analysis in the completion engine. Of course, if your type system is large or scoping rules are such that building symbol tables is expensive, consider optimizing by caching the symbol tables with the AST nodes.

---

A note about complexity

Things become complicated when you try to incrementally update that information, or avoid losing "expensive" state on "spurious states" (like, if the user enters only the opening part of a balanced scope, it would be wasteful to destroy the semantic information even though the user might immediately continue by closing the scope).

Be careful to plan ahead at which amount of complexity you are prepared to carry. It's okay to say "That's out of scope". In fact that's better than ending up with something like you commented on my very first response back then:

Well, MS Intellisense also is getting lost on invalid code, 20% cases is still up there ;-) [...] – PiotrK Nov 17 '17 at 19:40

I stand by my response, and I hope you realize this is basically the same response. You can elect to tackle arbitrary amounts of complexity if you have the resources/resolve. But always be sure you try to see just what you're starting, and where the buck stops.