missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'

Question

I am trying to write unit tests for spark scala code and I found this post: How to write unit tests in Spark 2.0+? However, when I add those dependencies I get this error when compiling:

Error:scalac: missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'.
Could not access type PlanTestBase in package org.apache.spark.sql.catalyst.plans,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.

I tried re-running with -Ylog-classpath but it did not help. This is what I think the relevant pom lines for the maven build:

<!-- #### SPARK #### -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.11.8</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.11.8</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.11</artifactId>
    <version>2.11.8</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>2.11.8</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
    <version>2.11.8</version>
</dependency>
<!-- #### SPARK #### -->

<!-- #### Testing #### -->
<dependency>
    <groupId>org.scalatest</groupId>
    <artifactId>scalatest_2.11</artifactId>
    <version>3.0.8</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.scalamock</groupId>
    <artifactId>scalamock_2.11</artifactId>
    <version>4.1.0</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.scalactic</groupId>
    <artifactId>scalactic_2.11</artifactId>
    <version>3.0.8</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.11.8</version>
    <type>test-jar</type>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.11.8</version>
    <type>test-jar</type>
    <scope>test</scope>
</dependency>
<!-- #### Testing #### -->

What could be causing this conflict?

score 0 · Answer 1 · answered Aug 06 '20 at 06:07

It is resolved by adding a spark-catalyst dependency. SQLTestUtilsBase trait extends PlanTestBase, and it's defined in the spark-catalyst package.

And also, I think you should change the spark version 2.11.8 to others in maven repo.

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-catalyst_2.11</artifactId>
    <version>2.4.0</version>
    <type>test-jar</type>
    <scope>test</scope>
</dependency>

see:

missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'

1 Answers1