0

I have developed a successful translator that uses ANTLR4 grammar and parse tree listeners. I'm very pleased with the rapid time to success using ANTLR for this project, and just started comparing binary outputs of my ANTLR4 based solution against a legacy C++ body of code that is slower. (My solution is targeted at speed so even though it's implemented in Java it could be faster).

However, when I started testing it with larger 110Mb input ASCII files, I find that I run out of HEAP. This occurs during the ANTLRInputStream instantiation. Which I believe I can fix with UnbufferedChar/Token streams. This stackoverflow question also suggests that the parse tree generation should be turned off, as the parse tree consumes a significant amount of memory.

If I turn off parse tree generation, my parse tree listeners won't be called. At least that's how I understand it. I suspect I won't be able to manage translating 1Gb files with the parse tree listener generation on. What is the solution?

I'd like to avoid moving my ParseTreeListener code to the grammar files if I can.

Community
  • 1
  • 1
Ross Youngblood
  • 502
  • 1
  • 3
  • 16
  • Read this Google Groups thread: https://groups.google.com/forum/#!topic/antlr-discussion/B06iG6Dht6w. It refers to a section in the ANTLR4 book on this topic (broadly) – Stephen C Feb 25 '17 at 01:37
  • How much memory are you allocating to the JVM? Specifying `-Xmx8g` should be sufficient to concurrently handle 1G of source files. Can you explain why you need to parse all of the source into a single parse tree? – GRosenberg Feb 25 '17 at 04:15
  • @GRosenberg Thanks! I'm building a huge tree as a newbie! Was looking at ways to manage/prune the tree, but your post makes me see that I can alter the grammar/parser (possibly) to parse in chunks...I need to consider this. May need some recursion behavior outside ANTLR. I'll look for examples of building/parsing smaller trees. Maybe it's in my ANTLR4 book and I missed it. But I think I can contrive some solution along these lines. I was pondering this late Friday. Found tree/prune flag in the ANTLR4 source, thought that might work too, but I like tree limiting better. – Ross Youngblood Feb 26 '17 at 08:06

0 Answers0