What are contexts in boost spirit X3?

Question

Short version

What are contexts (boost::spirit::x3::context) used for, what do they represent, why are they needed and not hidden to the end user, and what requirements should they meet for smooth compiling ?

Especially when having parsers in different translation units, for instance for unit testing.

Long version

In X3 documentation, when one want to understand precisely what is expected when a context is needed, it is not that clear.

Documentation says:

1. In the context of semantic actions :

from the context we can extract relevant information

2. Contexts are mentioned with BOOST_SPIRIT_DEFINE but nothing is said about it

3. Contexts are mentioned as well when explaining the expected program structures:

We'll also need to provide the initial context type. This is the context that X3 will use to initiate a parse. For calling phrase_parse, you will need the phrase_parse_context like we do below, passing in the skipper type

4. In the annotation tutorial contexts are again mentioned and manipulated, without clear explanation of what they are, and how they are expected to be used.

If you read the previous Program Structure tutorial where we separated various logical modules of the parser into separate cpp and header files, and you are wondering how to provide the context configuration information (see Config Section), we need to supplement the context like this:

5. And so on, the further we advance in the reading the more contexts are assumed to be known and the less they are explained.

So at the end, we pick one (the one provided in the documentation to start with)

x3::phrase_parse_context<x3::ascii::space_type>::type

and as long as it compiles...everything is fine.

Though quickly, compiler yells. That's not unusual: it's the compiler doing its job. But in this case it is hard to see how to get useful information from the error messages.

I see that the linker reports a missing symbol, but I need more information. It's hard to see where the symbol is used or how the types are inferred.

If only the docs told us what contexts are used for and what is really expected

I tried reading context.hpp which is not that long and quite easy to read. But it's so abstract that I still have no clue about what contexts model or represent.

And on SO, there are a few questions about contexts, mismatches, and excellent answers as well. I didn't really find out more about ask why contexts are needed, how they should be used, and when one needs to care about them. See e.g.

1. Mixing non-terminal rules from separeted translation unit
2. Embedding a parser from a separate translation unit into another parser
3. Splitting Boost.Spirit.X3 parsers into several TUs.

After having climbed the first hill of learning spirit/X3, I thought I had done the toughest part. That was without grasping the essence of contexts, which turns out important when linking.

Question

What are contexts (boost::spirit::x3::context) used for, what do they represent, why are they needed for the end user, and what requirements should they meet for smooth compiling ?

Answer 1

Contexts contains every bit of runtime state required for the parser to run and the following generic instantiation parameters:

• skipper type
• iterator type

Indeed, when splitting parsers across translation units, this is a frequent source of linker errors. Not in the least because

• as you note, contexts can be embellished by some directives
• contexts can sometimes contain full parser expression templates (see e.g. Mixing non-terminal rules from separeted translation unit) (*).

The documentation does basically state all these. But they focus on a happy path, and don't let you easily appreciate how cumbersome maintaining rules across translation units can become.

Playing Your Coach

I suggest a simple guideline

I will not spread/share rules beyond a single translation unit

such that their definitions can exist with their declaration/instantiations.

There are some downsides to that:

• the compilation time for that TU can be longer

• it might lead to big source file if the grammar is complicated

• lastly, due to effect observed above (*), there can be excessive template instantiations in simple scenarios because when local rules are recursively instantiating; they will lead to many "different" instantiations.

  This is exacerbated if parts of the grammar change e.g. skippers (e.g. using x3::skip/x3::no_skip).

Corollary:

If your grammar is really large, consider a standalone parser generator as opposed to Spirit

In spite of the limitations these all seem to imply, X3 is still a superb tool that enhances my productivity in a big way. I'm actively empowered to write small parsers properly and consistently.

PS/Asides

I think the linked answer should also help the broader questions about what contexts are. Beyond that, I could point you to a few [other] locations where I helped people debugging their linker errors in the context [sic] of X3.

• Recursive rule in Spirit.X3 (other directives that decorate the "state" - adding to the context: x3::with<>)

• Boost spirit x3 example calculator (calc8, calc9) linker error

• X3: Linker Error (unresolved external symbol "parse_rule") on nonterminal parser