Maven generates python code for python3 ANTLR grammar file

Question

I downloaded Python3 grammar file for ANLTR4 https://github.com/antlr/grammars-v4/blob/master/python/python3-py/Python3.g4 and when I run mvn generate-sources in eclipse, the Python3Lexer.java generated file contains some Java and some Python code eg.:

private void NEWLINE_action(RuleContext _localctx, int actionIndex) {
        switch (actionIndex) {
        case 0:

            tempt = Lexer.text.fget(self)
            newLine = re.sub("[^\r\n\f]+", "", tempt)
            spaces = re.sub("[\r\n\f]+", "", tempt)
            la_char = ""
            try:

antlr dependency in pom :

dependency>
            <groupId>org.antlr</groupId>
            <artifactId>antlr4-runtime</artifactId>
            <version>4.7.2</version>
            <exclusions>
                <exclusion>
                    <groupId>org.abego.treelayout</groupId>
                    <artifactId>org.abego.treelayout.core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

plugin>
                <groupId>org.antlr</groupId>
                <artifactId>antlr4-maven-plugin</artifactId>
                <version>4.7.1</version>
                <executions>
                    <execution>
                        <id>antlr</id>
                        <goals>
                            <goal>antlr4</goal>
                        </goals>
                        <configuration>
                            <listener>false</listener>
                            <visitor>true</visitor>
                            <treatWarningsAsErrors>true</treatWarningsAsErrors>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

any suggestions?

If you look inside the grammar file you linked, you'll see that it contains embedded Python code. If you want to generate Java code, your grammar file should contain embedded Java code or not contain any embedded code at all. — sepp2k, Feb 18 '21 at 23:30
Do note that the repository offers various versions of the Python 3 grammar for different languages, so you should pick the one for Java. — sepp2k, Feb 18 '21 at 23:32

Bart Kiers · Accepted Answer · 2021-02-19T15:27:55.350

There are some predicates inside the grammar that contain target specific code (Python in your case). You've picked this grammar:

https://github.com/antlr/grammars-v4/blob/master/python/python3-py/Python3.g4
                                                        ^^^^^^^^^^

but should take this one https://github.com/antlr/grammars-v4/blob/master/python/python3/Python3.g4 which contains Java code inside the predicates.

Just to make it clear, if your grammar looks like this:

grammar T;

parse
 : ANY* EOF
 ;

ANY
 : .
 ;

SPACE
 : [ \t\r\n] -> skip
 ;

ANTLR will generate any valid lexer and parser for you (Java. Python, C#, whatever valid target you want). However, if your grammar contains predicates:

grammar T;

parse
 : ( {_input.LT(1).getText().matches("\\d")}? ANY )* EOF
 ;

ANY
 : .
 ;

SPACE
 : [ \t\r\n] -> skip
 ;

then the part in between { and }? will simply be inserted inside the parser you're generating, regardless of the target (!).

That's why it's always advisable to not use predicate if possible. So in my example above, you could write the grammar like this instead of using a predicate:

grammar T;

parse
 : ANY* EOF
 ;

ANY
 : [0-9]
 ;

SPACE
 : [ \t\r\n] -> skip
 ;

(and then change ANY to DIGIT...)

Actually, the file `https://github.com/antlr/grammars-v4/blob/master/python/python3-py/pom.xml` is incorrect. Since the grammar contains Python action code, it must specify an element for `/project/build/plugins/plugin/configuration/arguments/argument=-Dlanguage=Python3`. And, there are two types of Python targets, so even this is wrong because "-py" is meaningless. I'm writing a generator for all target types which uses the pom.xml file or options+file queries, and using it to expand testing for all targets in the grammars-v4 repo. The test plugin is too primitive. — kaby76, Feb 19 '21 at 16:18

Maven generates python code for python3 ANTLR grammar file

1 Answers1