How to make fat jar in AWS KDA Flink Application?

Question

I need the following jar dependencies for my pyflink application.

1. flink-s3-fs-hadoop-1.15.2.jar
2. flink-sql-parquet-1.15.2.jar
3. flink-s3-fs-presto-1.15.2.jar

I want to package and deploy it to AWS Kinesis Data Analytics. AWS KDA needs one single fat jar for all the dependencies as described here

As a python developer, I am not much familiar how to package them. Any hints on how to create this fat jar from multiple jar dependencies?

Answer 1

There are many ways to create this fat (combined) jar. I recommend using the Apache Maven shade plugin.

This is the basic Maven configuration to construct the fat Jar.

<dependencies>
        <!-- Add your dependencies here -->
    </dependencies>
    <build>
        <plugins>
            <!-- The maven-shade plugin creates a fat jar that contains all
              dependencies. -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.4.1</version>
                <configuration>
                    <shadedArtifactAttached>true</shadedArtifactAttached>                       <shadedClassifierName>combined</shadedClassifierName> 
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Since you are a Python developer, I am assuming you are not familiar with Maven projects. Hence, I have created an open source project with installation instructions that you can use for constructing your fat jar.

https://github.com/shankarps/FatJarMaker

If you face issues, please report an issue on the GitHub project.