15

I have a java application, in which I'm using the Flink Api. So basically what I'm trying to do with the code, is to create two Datasets with few records and then register them as two tables along with the necessary fields.

 DataSet<Company> comp = env.fromElements(
                new Company("Aux", 1),
                new Company("Comp2", 2),
                new Company("Comp3", 3));

        DataSet<Employee> emp = env.fromElements(
                new Employee("Kula", 1),
                new Employee("Ish", 1),
                new Employee("Kula", 3));


        tEnv.registerDataSet("Employee", emp, "name, empId");
        tEnv.registerDataSet("Company", comp, "cName, empId");

And then I'm trying to join these two tables using the Table API:

Table anotherJoin = tEnv.sql("SELECT Employee.name, Employee.empId, Company.cName FROM " +
                "Employee RIGHT JOIN Company on Employee.empId = Company.empId");

And I'm just printing the results on the console. This works perfectly locally on my machine. I created a fat-jar by using the maven-shade-plugin with the dependencies and I'm trying to execute it in AWS Lambda.

So when I try to execute it there, I'm being thrown with the following exception (I'm posting only the first few lines):

reference.conf @ file:/var/task/reference.conf: 804: Could not resolve substitution to a value: ${akka.stream.materializer}: com.typesafe.config.ConfigException$UnresolvedSubstitution com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ file:/var/task/reference.conf: 804: Could not resolve substitution to a value: ${akka.stream.materializer} at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111) at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179) at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)

I extracted the jar before executing it in Lambda, and happened to see all the dependencies were there. I can't figure out where is it going wrong?

Any help could be appreciated.

Kulasangar
  • 9,046
  • 5
  • 51
  • 82

5 Answers5

19

You need to add this code in the pom -> maven-shaded-plugin -> configuration section:

<transformers>
    <!-- append default configs -->
    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
        <resource>reference.conf</resource>
    </transformer>
</transformers>
wind
  • 892
  • 1
  • 11
  • 27
  • I already had this in my pom, and didn't give me any luck. I'll anyways try it again and see. I guess the problem is that, aws lambda can't run **flink run the.jar** as we run in a local machine. – Kulasangar Feb 23 '18 at 06:25
  • 1
    Have you tried to run the fat-jar on your pc? I can also suggest to look into the fat jat and see what is inside the reference.conf file. – wind Feb 23 '18 at 07:15
  • Well no haven't tried running the fat jar locally, I'll try that first then. Ah yes please, so shall I link you up with the reference.conf? – Kulasangar Feb 23 '18 at 08:33
  • 1
    how about some description of what's going on here? – dtc May 24 '19 at 19:16
  • Combined the official doc https://ci.apache.org/projects/flink/flink-docs-stable/dev/projectsetup/dependencies.html#appendix-template-for-building-a-jar-with-dependencies and this answer. Worked! – madhairsilence Jan 06 '20 at 15:57
4

Was able to finally, figure this out, and it was some major version issues within my pom. I then downgraded all the dependencies to Flink 1.3.2 and also added <relocations> within the shade plugin. It works now. I'm attaching the entire pom, so that it would help someone someday:

<build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.7.0</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>


       <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.1.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>com.ink.FlinkLambdaTest.FlinkToLambda</mainClass>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                  <resource>reference.conf</resource>
                </transformer>
              </transformers>
              <relocations>
                <relocation>
                  <pattern>org.codehaus.plexus.util</pattern>
                  <shadedPattern>org.shaded.plexus.util</shadedPattern>
                  <excludes>
                    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                  </excludes>
                </relocation>
              </relocations>
            </configuration>
          </execution>
        </executions>
      </plugin>  
    </plugins>
  </build>

<dependencies>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-table_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-java</artifactId>
      <version>1.3.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-clients_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-scala_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-scala_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    </dependencies>

Make sure to change the main class with yours.

Kulasangar
  • 9,046
  • 5
  • 51
  • 82
1

In gradle:

buildscript {
    repositories {
        maven {
            url 'https://plugins.gradle.org/m2/'
        }
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
    }
}
apply plugin: 'com.github.johnrengelman.shadow'

Then, in the project(s) that aggregate the akka dependcies:

shadowJar {
    include 'reference.conf'
}

With this in place you should be able to build normally (e.g. by calling the assemble and build tasks).

Intellij IDEA supported this, as well. FTW! It also supported the SBT construct, below.

lazy val assemblySettings = Seq(
  assemblyJarName in assembly := name.value + ".jar",
  assemblyMergeStrategy in assembly := {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case _ => MergeStrategy.first
  })
F. P. Freely
  • 1,026
  • 14
  • 24
  • 4
    The answer at https://stackoverflow.com/questions/34326168/how-can-i-fix-missing-conf-files-when-using-shadowjar-and-scala-dependencies/34326169#34326169 worked much better for me – andrewdotn Jul 24 '18 at 22:58
1

For SBT users, this works for me.

assemblyMergeStrategy in assembly := {
  {
    case PathList("META-INF", xs@_*) => MergeStrategy.discard
    case "reference.conf" => MergeStrategy.concat
    case x => MergeStrategy.first
  }
}
Wei Chen
  • 605
  • 6
  • 14
0

This might come a little bit late, but here is the answer from the flink website concerning this topic.

Also information about the META-INF folder are excluded, which might cause security problems when using the jar.

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.0.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <artifactSet>
                            <excludes>
                                <exclude>com.google.code.findbugs:jsr305</exclude>
                                <exclude>org.slf4j:*</exclude>
                                <exclude>log4j:*</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <!-- Do not copy the signatures in the META-INF folder.
                                Otherwise, this might cause SecurityExceptions when using the JAR. -->
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>my.programs.main.clazz</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
User12547645
  • 6,955
  • 3
  • 38
  • 69