6

I'm writing a custom flex file to generate a lexer for use with JSyntaxpane.

The custom language I need to lex has different states that can be embedded into each other in a kind of stack.

I.E you could be writing an expression that has a single quoted string in it and then embed another expression within the string using a special token eval(). But you can also embed the expression within a double quoted string.

eg:

someExpressionFunction('a single-quoted string with an eval(expression) embedded in it', "a double-quoted string with an eval(expression) embedded in it")

This is a simplification, there are more states than this, but assuming I need to have different states for DOUBLE_STRING and SINGLE_STRING it adequately describes my situation.

What's the best way to ensure I return to the correct state upon closing the eval expression (i.e return to DOUBLE_STRING if I was in double quotes, SINGLE_STRING if I was in single quotes)

The solution I've come up with, which works, is to keep track of state using a Stack and some custom methods to use in lieu of using yybegin to start a different state.

private Stack<Integer> stack = new Stack<Integer>();

public void yypushState(int newState) {
  stack.push(yystate());
  yybegin(newState);
}

public void yypopState() {
  yybegin(stack.pop());
}

Is this the best way to achieve this? Is there a simpler built-in function of JFlex I can leverage or a best practice I should know about?

Tom Martin
  • 2,498
  • 3
  • 29
  • 37
  • 1
    Came up with same idea, but I thought it's bad practice. So now it's much easier to implement embedded states! – Dany Dec 15 '13 at 19:53

1 Answers1

5

I think that's one very good way of doing it. I actually needed some similar feature to add Groovy GString, Python like String and some HTML to JavaDocs.

What I would also like to add is a Lexer calling a Lexer to parse sub sections. Something like JavaScript embedded in HTML. But I could not get the time to do it.

I like StackOverflow, but just wondering why didn't you post this on JSyntaxPane's issues?

Ayman
  • 11,265
  • 16
  • 66
  • 92
  • Hi Ayman, sorry I've only just seen your answer. I also like StackOverflow so I guess I was asking just to see if there was much of flex/lexing/parsing community here. I guess I should have cross posted at jsyntaxpane. I have posted there in the past (I'm TomPoges there). Nice work on JSyntaxPane btw. – Tom Martin Sep 22 '09 at 08:11