Spark 2.1 with Pureconfig 0.8 workaround for Maven

Question

The below solution taken from Spark not working with pureconfig seems to be the working solution for sbt but having a hard time figuring out a maven version for doing this. Trying to get pureconfig 0.8 working with spark 2.1 using spark-submit but still getting the pesky Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness; error outside of IntelliJ.

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("shapeless.**" -> "shadeshapless.@1")
    .inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
    .inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0")
    .inProject
)

Have also tried proposed solution from here Spark with Pureconfig - proper maven shade plugin configuration but still no luck.

This is the final configuration that has worked if I use the uber jar that gets created but I'm not sure I fully understand how maven shading works and is there a way to avoid having to create an additional renamed jar? Ideally I want to just use the jar with dependencies that gets created and not create an additional third jar with the below:

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-shade-plugin</artifactId>
       <version>3.0.0</version>
       <executions>
           <execution>
               <phase>package</phase>
               <goals>
                   <goal>shade</goal>
               </goals>
           </execution>
       </executions>
       <configuration>
           <createDependencyReducedPom>false</createDependencyReducedPom>
           <relocations>
               <relocation>
                   <pattern>shapeless</pattern>
                   <shadedPattern>com.shaded.shapeless</shadedPattern>
               </relocation>
           </relocations>
           <filters>
               <filter>
                   <artifact>*:*</artifact>
                   <excludes>
                       <exclude>META-INF/*.SF</exclude>
                       <exclude>META-INF/*.DSA</exclude>
                       <exclude>META-INF/*.RSA</exclude>
                   </excludes>
               </filter>
           </filters>
           <finalName>uber-${project.artifactId}-${project.version}</finalName>
       </configuration>
   </plugin>

If your getting an exception with the shapeless package name that means that shading your source code or some dependency inside it isn’t working properly. Can you add the shading configuration in Maven? — Yuval Itzchakov, Dec 09 '17 at 17:06
thank you. updated question to add the configurations I have tried. — horatio1701d, Dec 09 '17 at 18:24

score 0 · Answer 1 · answered Dec 12 '17 at 23:06

Sorry for answering so late. I'm really not sure what's causing the problem. I changed my pureconfig version to 0.8.0 (was using 0.7.2) and everything still seems to work.

As it does not seem easy, I'll try to give you more insight about how my project is structured, maybe this will somehow help. Oh and btw, I'm using maven 3.3.9 but I doubt it matters (who knows though).

So, my project consists of submodules, i.e. I have the top level pom where <modules></modules> are specified, and every module has its own pom.

Now, in the top level pom, I use dependency management as well as plugin management, so it looks like this:

</dependencyManagement>
  <dependencies>
    <dependency>
        <!-- some dependency -->
    </dependency>
  </dependencies>
</dependencyManagement>

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.0.0</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <relocations>
                        <relocation>
                            <pattern>shapeless</pattern>
                            <shadedPattern>com.matek.chuusai.shapeless</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

Now in the submodules, configuring the dependencies as well as adding maven shade plugin looks like this:

    <dependencies>
      <dependency>
          <!-- some dependency -->
      </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-shade-plugin</artifactId>
                <groupId>org.apache.maven.plugins</groupId>
            </plugin>
        </plugins>
    </build>

If your project has submodules, try to configure it this way. Hope this helps (somehow).

Spark 2.1 with Pureconfig 0.8 workaround for Maven

1 Answers1

Linked