I am new to intellij plugin writing. I started writing a intellij plugin for one of our custom language. I am following the tutorial given in intellij official site. Also I downloaded their Grammar-Kit repo from github to understand the code base.
While writing my plugin I am using jflex for Lexer and bnf grammar for parser. I find difficulty to implement a lexer to send proper tokens to the parser.
What I see in the Grammar-Kit repo that Lexer (for bnf) is pretty simple and it is only recognizing String, Number, Id, white space, comments and special characters('(', '*' etc.). It does not recognize bnf keywords like 'private', 'external', 'meta' etc.
Now when I am seeing the PSI tree of a example bnf file say for the line,
private myRule ::= '(' myExpression ')' ';'
is as below,
BnfFile:Dummy.bnf(0,43)
BNF_RULE:myRule(0,43)
BNF_MODIFIER(0,7)
PsiElement(id)('private')(0,7)
PsiWhiteSpace(' ')(7,8)
PsiElement(id)('myRule')(8,14)
PsiWhiteSpace(' ')(14,15)
PsiElement(::=)('::=')(15,18)
PsiWhiteSpace(' ')(18,19)
BNF_SEQUENCE: '(' myExpression ')' ';'(19,43)
BNF_STRING_LITERAL_EXPRESSION: '('(19,22)
PsiElement(string)(''('')(19,22)
PsiWhiteSpace(' ')(22,23)
BNF_REFERENCE_OR_TOKEN: myExpression(23,35)
PsiElement(id)('myExpression')(23,35)
PsiWhiteSpace(' ')(35,36)
BNF_STRING_LITERAL_EXPRESSION: ')'(36,39)
PsiElement(string)('')'')(36,39)
PsiWhiteSpace(' ')(39,40)
BNF_STRING_LITERAL_EXPRESSION: ';'(40,43)
PsiElement(string)('';'')(40,43)
What i am seeing here is 'private' is recognized as PsiElement.id
by the Lexer, after that some code wrap it with a BnfModifier object which is declared the bnf file. The same for BNF_SEQUENCE, BNF_REFERENCE_OR_TOKEN, BNF_STRING_LITERAL_EXPRESSION etc. They are all not recognized by Lexer, however some code wrap them with bnf file declared objects.
I want to understand how this wrapping is done on top of Lexer recognized tokens. It will help me to recognize keyword in our DSL and highlight them differently, auto-complete them etc.
Thanks,
Subhojit