Spark Maven dependency incompatibility between delta-core and spark-avro

Question

I'm trying to add delta-core to my scala Spark project, running 2.4.4.

A weird behaviour I'm seeing is that it seems to be in conflict with spark avro. Maven build succeeds, but during runtime I'm getting errors.

If delta table dependency is declared first, I get a runtime error that spark avro is not installed:

User class threw exception: org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of "Apache Avro Data Source Guide".;

<dependencies>
    <dependency>
        <groupId>io.delta</groupId>
        <artifactId>delta-core_2.11</artifactId>
        <version>0.6.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-avro_2.11</artifactId>
        <version>2.4.4</version>
    </dependency>

if spark avro is defined first, Avro works, but delta gets an exception:

User class threw exception: java.lang.ClassNotFoundException: Failed to find data source: delta. Please find packages at http://spark.apache.org/third-party-projects.html

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-avro_2.11</artifactId>
            <version>2.4.4</version>
        </dependency>

        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-core_2.11</artifactId>
            <version>0.6.1</version>
        </dependency>

I thought this could be some kind of dependency conflict so I tried:

 <exclusions>
        <exclusion>
            <groupId>*</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>

on both, but it didn't help.

score 0 · Answer 1 · answered Aug 18 '20 at 21:06

Got an answer for this in the delta-core issues page. Thanks zsxwing!

The complete solution is based on this previous stack overflow answer to merge the services under META-INF so the different spark sources wont override each-other.

The complete solution - I changed the maven assembly plugin to this:

<plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.3.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <descriptors>
                        <descriptor>${project.basedir}\src\assembly\tvm_assembly.xml</descriptor>
                    </descriptors>
                </configuration>
            </plugin>

And in the new tvm_assembly.xml file (based on the original jar-with-depndencies, added the properties for merge):

<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.1.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.1.0 http://maven.apache.org/xsd/assembly-2.1.0.xsd">
    <!-- TODO: a jarjar format would be better -->
    <id>jar-with-dependencies</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>true</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
    <containerDescriptorHandlers>
        <containerDescriptorHandler>
            <handlerName>metaInf-services</handlerName>
        </containerDescriptorHandler>
        <containerDescriptorHandler>
            <handlerName>metaInf-spring</handlerName>
        </containerDescriptorHandler>
        <containerDescriptorHandler>
            <handlerName>plexus</handlerName>
        </containerDescriptorHandler>
    </containerDescriptorHandlers>
</assembly>

Spark Maven dependency incompatibility between delta-core and spark-avro

1 Answers1