1

I have a very long antlr4 grammar file, and recently I got into an error looking like this

Exception in thread "main" java.lang.UnsupportedOperationException: Serialized ATN data element 68939 element 11 out of range 0..65535
    at org.antlr.v4.runtime.atn.ATNSerializer.serialize(ATNSerializer.java:361)
    at org.antlr.v4.runtime.atn.ATNSerializer.getSerialized(ATNSerializer.java:601)
    at org.antlr.v4.Tool.generateInterpreterData(Tool.java:745)
    at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:400)
    at org.antlr.v4.Tool.process(Tool.java:361)
    at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:328)
    at org.antlr.v4.Tool.main(Tool.java:172)

I have seen previous questions and it turns out this is a memory issue. I have also done a full debugging exercise and reduced the grammar until this error disappears multiple times with multiple parts of the grammar, and there doesn't appear to be any specific bit of the grammar code which is causing the error, it's just the size. As soon as I reduce the grammar below a certain size, no matter what I remove, it starts working again. So my question is: is there any way around this issue within antlr4? For example, using a computer with bigger RAM memory, specify a setting when running antlr4 that increases the max allowed grammar size, etc., or is reducing the grammar the only possible solution (short of not using antlr4 and trying a completely different parser)?

andrea
  • 482
  • 5
  • 22
  • 1
    As commented under my (removed) answer: it is not a memory issue looking at the error: `Serialized ATN data element 68939 element 11 out of range 0..65535` – Bart Kiers Oct 22 '21 at 12:06
  • I see. Do you know what the issue actually is then? The error seems a bit vague and I'm not sure what the range up to 65535 is referring to. Most importantly: do you know if there is a way to avoid it or go around it, short of reducing the grammar size? Or is this size limit just ingrained in antlr4 such that there is no solution to this? Thanks a lot! – andrea Oct 22 '21 at 12:36
  • 1
    Reduce the amount of keywords: https://github.com/antlr/antlr4/issues/1863 . AFAIK, there is no predefined amount of keywords. But the max is usually never reached when using ANTLR as it's "supposed" to be used. I bet you're in a similar situation as the person that opened the issue in Github where the amount of keywords was absurdly large. To get more specific answers, share you grammar. – Bart Kiers Oct 22 '21 at 12:58
  • By the look of the issue it seems I have exactly the same problem. Unfortunately I can't share the grammar, but it was used in a way which is a bit outside the usual purpose of a grammar, hence the thousands of keywords. At least I now know that there is a memory limit within antlr4 and that there is nothing to do about it. Many thanks for confirming. – andrea Oct 23 '21 at 20:29

0 Answers0