Linkage failure when running Apache Flink jobs

Question

I have a job developed in Flink 0.9 that is using the graph module (Gelly). The job is running successfully within the IDE (Eclipse) but after exporting it to a JAR using maven (mvn clean install) it fails to execute on the local flink instance with the following error

"The program's entry point class 'myclass' could not be loaded due to a linkage failure"

java.lang.NoClassDefFoundError: org/apache/flink/graph/GraphAlgorithm

Any idea why is this happening and how to solve it?

Did you create the project pom using Flink's quickstart.sh (=Maven archetype) ? Do you have `flink-gelly` as a dependency in your pom? — Robert Metzger, May 07 '15 at 13:56

score 4 · Accepted Answer · answered May 13 '15 at 13:03

It looks like the code of flink-gelly did not end up in your jar file. The most obvious reason for this issue is the missing maven dependency in your project's pom file. But I assume the dependency is present, otherwise developing the job in the IDE would be impossible.

Most likely, the jar file has been created by the maven-jar-plugin, which is not including dependencies. Try adding the following fragment to your pom.xml:

    <build>
    <plugins>
        <!-- We use the maven-shade plugin to create a fat jar that contains all dependencies
        except flink and it's transitive dependencies. The resulting fat-jar can be executed
        on a cluster. Change the value of Program-Class if your program entry point changes. -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.3</version>
            <executions>
                <!-- Run shade goal on package phase -->
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <filters>
                            <filter>
                                <artifact>org.apache.flink:*</artifact>
                                <excludes>
                                    <exclude>org/apache/flink/shaded/**</exclude>
                                    <exclude>web-docs/**</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <!-- add Main-Class to manifest file -->
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>YOURMAINCLASS</mainClass>
                            </transformer>
                        </transformers>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>

</build>
<profiles>
    <profile>
        <!-- A profile that does everyting correctly:
        We set the Flink dependencies to provided -->
        <id>build-jar</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <dependencies>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-java</artifactId>
                <version>0.9-SNAPSHOT</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-streaming-core</artifactId>
                <version>0.9-SNAPSHOT</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.flink</groupId>
                <artifactId>flink-clients</artifactId>
                <version>0.9-SNAPSHOT</version>
                <scope>provided</scope>
            </dependency>
        </dependencies>
    </profile>
</profiles>

Now, you can build the jar using mvn clean package -Pbuild-jar. The jar file will now be located in the target/ directory.

You can manually check whether the jar (zip) file contains class files in /org/apache/flink/graph/

That is true. I had to remove the maven-jar-plugin from the pom.xml and add the above snippet after modifying the Flink verison. I can run it now on the local instance successfully — Karim Wadie, May 14 '15 at 02:01

Linkage failure when running Apache Flink jobs

1 Answers1