Make a table containing tokens visible for both .mly and .mll by menhir

Question

I would like to define a keyword_table which maps some strings to some tokens, and I would like to make this table visible for both parser.mly and lexer.mll.

It seems that the table has to be defined in parser.mly,

%{ 
  open Utility (* where hash_table is defined to make a table from a list *)
  let keyword_table = hash_table [
      "Call", CALL; "Case", CASE; "Close", CLOSE; "Const", CONST; 
      "Declare", DECLARE; "DefBool", DEFBOOL; "DefByte", DEFBYTE ]
%}

However, I could NOT use it in lexer.mll, for instance

{
open Parser
let x = keyword_table (* doesn't work *)
let x = Parser.keyword_table (* doesn't work *)
let x = Parsing.keyword_table (* doesn't work *)
}

As this comment suggests, menhir has a solution for this, could anyone tell me any details?

Answer 1

The first option is to define tokens in a separate .mly file. Executing menhir for this file with --only-tokens option will generate a module containing type token that you can use in your parser compiled with --external-tokens option.

If this solves the problem with tokens, you can specify all other functions that are used by both parser and lexer in a separate file as Thomash suggested.

There is an alternative solution as well. You can use %parameter<module signature> declaration in the parser to parametrize the entire parser over type and function annotations specified inside given signature. The main advantage is that this signature is provided in the interface file for the parser, so the parser can share this signature with other modules (that can construct modules based on the signature).

I suggest to refer to menhir examples, namely see calc-two to get know about external tokens and to calc-param to know how to create parametrized parsers.

Answer 2

I usually put the keyword_tablein lexer.mll and I see no reason to put it in parser.mly. If you need to access it from both lexer.mll and parser.mly (but why do you want to access it from parser.mly?), the easiest solution is to put it in a third file keyword.ml and use Keyword.keyword_table (or open Keyword and keyword_table).