1

I have exactly the same problem as described here: Spark not working with pureconfig. The one and only answer to the question above seems reasonable, but I am working with Maven instead of sbt and I'm failing to translate the posted solution from sbt to Maven.

I have tried something like the following:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
    <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
    </execution>
</executions>
<configuration>
    <createDependencyReducedPom>false</createDependencyReducedPom>
    <relocations>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.3.2</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.0.0</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.0.0</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.github.pureconfig</pattern>
            <shadedPattern>com.matek.shaded.com.github.pureconfig</shadedPattern>
            <excludes>
                <exclude>com.chuusai:shapeless_2.11:2.3.2</exclude>
            </excludes>
            <includes>
                <include>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</include>
            </includes>
        </relocation>
    </relocations>
</configuration>

But not surprisingly, this does not work (I am not even sure whether it is correct). How to specify the maven shade plugin configuration to make it work with spark submit?

Matek
  • 641
  • 5
  • 16

1 Answers1

5

I managed to solve the issue. It was actually my mistake, the pattern is simply shapeless and not anything like com.chuusai.shapeless. This worked:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <relocations>
                        <relocation>
                            <pattern>shapeless</pattern>
                            <shadedPattern>com.matek.shaded.shapeless</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </plugin>
Matek
  • 641
  • 5
  • 16
  • Unfortunately, not working for me: https://github.com/pureconfig/pureconfig/issues/333 – Georg Heiler Dec 04 '17 at 15:04
  • Same here. have not found a working example for maven and using spark 2.1. – horatio1701d Dec 09 '17 at 11:41
  • Just to mention the GeorgHeiler comments, it seems he managed to get it to work, so @horatio1701d please check the github link. I'm actually doing it with spark 2.1 and it works, so additionaly make sure that you have all the package names correct (that's what I've been struggling the most with). – Matek Dec 09 '17 at 18:23
  • @Matek, do you have a working maven configuration you can share? the link seems to only be for sbt unless I'm missing something. I tried your maven configuration proposed but that gives me same issue. Can you be more specific which package names I should be careful with in my maven configuration perhaps. tried using exactly what you added after also adding shapeless maven package 2.3.2. Not sure what I'm missing. – horatio1701d Dec 09 '17 at 18:28
  • https://stackoverflow.com/questions/47728372/spark-2-1-with-pureconfig-0-8-workaround-for-maven?noredirect=1#comment82422446_47728372 – horatio1701d Dec 10 '17 at 13:07