3

My Lexer is supposed to distinguish brackets and maintain a stack of opened brackets during lexing. For this I specified a helper function in my fsl file like this:

let updateBracketStack sign =  // whenever a bracket is parsed, update the stack accordingly
    match sign with
    | '[' -> push sign
    | '{' -> push sign
    | ']' -> if top() = '[' then pop() else ()
    | '}' -> if top() = '{' then pop() else ()
    | _ -> ()

The stack of course is a ref of char list. And push, top, pop are implemented accordingly.

The problem is that everything worked up until I added the { character. Now FsLex simply dies with error: parse error

If I change the characters to strings, i.e. write "{" FsLex is fine again, so a workaround would be to change the implementation to a stack of strings instead of characters.

My question is however, where does this behaviour come from? Is this a bug if FsLex?

Friedrich Gretz
  • 535
  • 4
  • 14
  • 1
    The code you've posted has `top()`'s result being compared to `'['` (a char) in one match branch, and `"{"` (a string) in another match branch. That shouldn't compile! The return type of `top()` can either be `char` or `string`, but either way, the F# compiler should be giving you a type error on one of those two branches. Are you sure you haven't made a copy-and-paste error somewhere in pasting in your code for this question? – rmunn Nov 11 '16 at 16:29
  • Fixed the typo, thanks. However this is unrelated to the question. It is not the F# compiler not compiling but fslex is failing to accept the input and generate any F# code. – Friedrich Gretz Nov 14 '16 at 16:33

1 Answers1

0

FsLex's parser is generated using FsLexYacc. The message "parse error" means the lexing (of your .fsl file) until error position is succeeded but parsing is failed at that position. To find the root cause you will need to post full input text to FsLex.

This is only guess. FsLex could be confused by '{' character since it is also open token for embedded code block? Or your input text contains some special unicode characters but it looks like whitespace on editor?

One possible workaround is, to create another module and .fs file, LexHelper module in LexHelper.fs, and place your helper functions in it, and open it from .fsl file.

EDIT

Looking at the source code of FsLexYacc, it doesn't handle } character enclosed by single-quotes in embedded F# code, but does when enclosed by double-quotes.

https://github.com/fsprojects/FsLexYacc/blob/master/src/FsLex/fslexlex.fsl