1

I am currently working on creating a DSL in xText and I am stumbling upon a problem which is probably related to an ambiguity or left recursion problem. I am not sure which of these two problems applies to my case (similar topics I found online also mention that these problems often seem related) but I guess it has to do with left recursion.

On line 100 in my DSL code I declare a rule called Expression. As you can see it aggregates multiple other types (which on their part again aggregate multiple other types and eventually types called Factor (on line 130) can also be aggregated). Eventually this whole 'aggregation tree' boils down to a problem with this Factor type. As you can see, this type can aggregate an Expression again. So there is a loop; an Expression type can eventually contain a Factor type, and the Factor type can then again contain an Expression type (after which this loop can theoretically continue infinitely; I guess that's where the problem is because the ANTLR parser used by xText was not designed for this kind of recursion). I tried to solve this problem by using a syntactic predicate (=> symbol) in the Expression type (see

(=> "endSimpleExpression")

but it's still not working. I know for sure that it has to do with the relationship between the types Expressions and Factor (because if I don't add Expression types in the Factor type, the DSL works just fine). I assume that I am not placing the syntactic predicate on the right place. Another solution that I considered was the use of left Factoring, but I don't know how to apply left factoring in this case. I am curious to your thoughts on this problem.

grammar org.xtext.example.mydsl.FinalDsl with org.eclipse.xtext.common.Terminals

generate finalDsl "http://www.xtext.org/example/mydsl/FinalDsl"


Model:
    'functionName' name = STRING
    functions += FunctionElements*
;

// Function elements of which the model exists. The model can contain
// library functions, for loops, and if/else statements.
  FunctionElements:
    ifElseStatements += IfElseStatements |
    statements += Statement 
; 

// IfElse Statements requiring if statements and optionally followed by
// one else statement.
IfElseStatements: 
    ifStatements += IfStatements
    (elseStatement = ElseStatement)?
;

// If statements requiring conditions and optionally followed by
// library functions or for loops.
IfStatements:
    'if'
    expression = Expression
    (ifFunctions += libraryFunctionsEnum | forLoops += ForLoops)
;

// Else statement requiring one or multiple library functions.
ElseStatement:
    'else' elseFunctions += libraryFunctionsEnum
;

// For loops requiring one condition and followed by zero or more
// library functions
ForLoops:
    'for'
    expressions = Expression
    libraryFunctions += libraryFunctionsEnum*

;


Statement:
    //compoundStatement += CompoundStatement | //left out of Statement because 
    // otherwise a recursive call exists (statement += compoundstatement += statement
    simpleStatement += SimpleStatement |
    structuredStatement += StructuredStatement
;

SimpleStatement:
    classOperationStatement += ClassOperationStatement | 
    libraryInterFaceMethodStatement += LibraryInterFaceMethodStatement | 
    libraryPersistenceMethodStatement += LibraryPersistenceMethodStatement
;

StructuredStatement:
    forLoops += ForLoops | ifElseStatements += IfElseStatements
;



ClassOperationStatement:
    classOperationName += libraryFunctionsEnum
;

LibraryInterFaceMethodStatement:
    interfaceMethods += libraryInterFaceMethodStatementEnum
;


LibraryPersistenceMethodStatement:
    persistenceMethods += libraryPersistenceMethodStatementEnum
;




//*Eventually filled with details from class diagram, but for now we manually fill it for the sake of testing.
enum libraryFunctionsEnum:
    login='login'|
    hasCode= 'encrypt'|
    display='display'
;

enum libraryPersistenceMethodStatementEnum:
    createInstance = "createInstance" |
    log = "log"
;

enum libraryInterFaceMethodStatementEnum:
    mesasge = "message" |
    error = "error"
;

Expression:
simpleExpression = SimpleExpression 
(relationalOperator = RelationalOperator 
additionalSimpleExpression = SimpleExpression)?
(=> "endSimpleExpression")
;


SimpleExpression:

    term = Term
    additionalExpressions += AdditionalExpressions*
;

AdditionalExpressions:
    additionOperator = AdditionOperator
    term = Term
;

Term:
    factorTerm = Factor
    additionalTerm += AdditionalTerm*
;

AdditionalTerm:
    multiplicationOperator = MultiplicationOperator 
    factor = Factor
;

// We can optionally integrate Java types right here (int, boolean, string, etc.)
Factor: {Factor} (
    "("  expression = Expression ")" |
    //'not' factor += Factor |
     operationParameterName = OperationParameterName |
    classAttributeName += ClassAttributeName |
     INT //| STRING //| set = Set 
    )
;

OperationParameterName: // We can use identifiers right here, but for now I put in a string
    'operationParameter' STRING
;

ClassAttributeName: // We can use identifiers right here, but for now I put in a string
    STRING
;

RelationalOperator:
"=" | "<>" | "<" | "<=" | ">" | ">=" | "in"
;

AdditionOperator:
"+" | "-" | "or"
;

MultiplicationOperator:
"*" | "/" | "and"
;

enum logicalOperators:
    greaterThan='>'|
    smallerThan='<'|
    greaterOrEqualThan='=>'|
    smallerOrEqualThan='<='|
    equalTo='=='
;
Jos B.
  • 27
  • 5
  • There is nothing wrong with the fact that an expression can contain a term and a term can contain an expression. Not only is ANTLR perfectly capable of handling that kind of recursion, but you need that kind of recursion for most languages. What ANTLR can't handle is left recursion (though ANTLR 4 can handle it if you structure your grammar in a certain way, but that's irrelevant because xtext uses ANTLR 3), but your rule isn't left recursive because there's a "(" before the call to `Expression`, meaning the recursion isn't the left-most part of the definition. – sepp2k May 09 '19 at 16:38
  • So please be more specific about your problem. Do you get an error message that is telling you that your grammar is ambiguous or left recursive? If so please post that error message exactly. Or does the grammar simply not match what you want it to match? If so, please show a simple input that should match, but doesn't (or vice versa). – sepp2k May 09 '19 at 16:41
  • @sepp2k thanks for your reply. The problem is not that my xText grammar doesn't compile. The above grammar actually does not give any direct errors, but when it compiles I can't execute the compiled files (well I can, but when executing the compiled files to start my DSL in a new Eclipse Environment, a plugin error appears meaning that apparently something went wrong). The console does however state the following error when compiling the grammar: – Jos B. May 09 '19 at 22:06
  • error(211): ../org.xtext.example.finaldsl/src-gen/org/xtext/example/mydsl/parser/antlr/internal/InternalFinalDsl.g:139:2: [fatal] rule ruleFunctionElements has non-LL(*) decision due to recursive rule invocations reachable from alts 1,2. Resolve by left-factoring or using syntactic predicates or using backtrack=true option. – Jos B. May 09 '19 at 22:06
  • So 'under the hood' this does seem to be a left-recursion problem (despite the fact that it starts with "(" as you have already noted). The thing is that when I remove these brackets, the rules which are impacted by this left-recursion problem are immediately underlined with a red line. When hovering my mouse over them, it states 'This rule call is part of a left recursive call graph'. So it does seem to be a left recursion problem (doesn't matter if you add "(" brackets or not). – Jos B. May 09 '19 at 22:13

1 Answers1

0

InternalFinalDsl.g:139:2: [fatal] rule ruleFunctionElements has non-LL(*) decision due to recursive rule invocations reachable from alts 1,2.

So let's look at the rule FunctionElements:

  FunctionElements:
    ifElseStatements += IfElseStatements |
    statements += Statement 
;

Okay, so FunctionElements can either be an IfElseStatement or a Statement. That sounds suspicious and sure enough: a Statement can be a StructuredStatement, which can in turn be an IfElseStatement. So the above is ambiguous because an IfElseStatement could either be derived directly via FunctionElements -> IfElseStatement or indirectly via FunctionElements -> Statement -> StructuredStatement -> IfElseStatement.

So you should simply remove the IfElseStatement alternative because it is redundant.

sepp2k
  • 363,768
  • 54
  • 674
  • 675
  • Thanks again. That makes total sense now that you state it (I probably left it there accidentally while further elaborating my DSL). I have stated the FunctionElements rule as follows now (so it does not have redundant IfElseStatements). `FunctionElements: statements += Statement ; ` However, the problem with the Expression and Factor type ending up in a left recursive call graph problem is unfortunately still there. I found another [topic](https://www.eclipse.org/forums/index.php/t/1066520/) where a problem which seems similar to mine is discussed. – Jos B. May 09 '19 at 23:15
  • If you look at the last comment of Christian Dietrich in that topic, he proposes a grammar which seems to solve the problem discussed over there. The last three rules of that grammar seem to solve problem, by defining the expression as containing a left and right part (look at the rule called PARLIST where he defines this). I am looking now on how to translate this solution to my DSL. – Jos B. May 09 '19 at 23:27
  • @JosB. What exactly do you mean when you say that you "the problem with the Expression and Factor type ending up in a left recursive call graph problem is unfortunately still there"? You've said that you get an error about left recursion when you remove the "(", which makes sense because that change will make your grammar left-recursive. You've also said that the error goes away if you put the "(" back. So as far as I'm concerned that's the end of any left-recursion problems you have. – sepp2k May 09 '19 at 23:33
  • If you get another error about left-recursion somewhere else in the grammar, please post that error. If not, you don't have any left recursion and any problems you do have are unrelated to left recursion. – sepp2k May 09 '19 at 23:33
  • Note that the grammar in the thread you've linked is actually left recursive and doesn't seem to have anything to do with your grammar. So I don't think that thread is relevant to your problem. – sepp2k May 09 '19 at 23:34
  • Nice it's working. If I add the "(" parentheses the DSL works. I'm quite new to xText (and I just discovered left recursion today because of this problem) so as you may have noticed I am trying to learn all the ins and outs of xText and its concepts. It would be nice if I could apply left refactoring to this grammar (so that I can also add expressions without the "(" parentheses), but that's something I won't bother you with (unless it's something very easy to explain for you). Thanks anyway for making my day and providing responses so rapidly. Votes and karma coming your way ;) – Jos B. May 09 '19 at 23:52
  • My level on Stackoverflow is still < 15 so unfortunately I can't give you any public votes for this answer, but I would like to thank you a lot!! – Jos B. May 09 '19 at 23:54
  • @JosB. Left factoring doesn't fix left-recursion, it fixes situations where the parser can't decide which alternative to take without more lookahread. In order to work around left-recursion, you'll want to rewrite your rules using repetitions (like `A: B (op B)*;` instead of `A: B | A op B;`, but that doesn't really apply to your `Factor: '(' Expression ')'` rule. That rule would simply isn't left-recursive and would be wrong if it were (that is, `Factor: Expression` would be a pointless rule that would make your grammar ambiguous and wouldn't allow parentheses in the language). – sepp2k May 09 '19 at 23:58
  • thanks for your elaboration. I think I get the idea now (adding the '(' keyword to the Factor rule makes it clear whether an Expression is inside a Factor or not. If this keyword is not placed there, it's unclear (ambigious) what Expression we are talking about (one that's nested in a Factor or one that's not nested). I was confused because the error was highlighted as a left recursive call graph error. Seems like I still have a lot to learn. Thanks!! – Jos B. May 10 '19 at 08:48