scala spark version mismatch

Question

i am getting below exception while running word count program in scala.

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
    at org.apache.spark.util.Utils$.getCallSite(Utils.scala:1406)

googling for the resolution i can understand this happens when there is a mismatch between spark and scala. my pam dependeny is

   <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>Test</groupId>
    <artifactId>Test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>

    </dependencies>

</project>

and the scala version in project settings scala-sdk-2.11.8

Not sure what is wrong here.i have spent considerable amount of time in maven repository trying with different version combinations.

from my local spark installation i figured out the right scala version by running command scala.util.Properties.versionString

i have selected the same scala sdk for the project.but no luck.

your help is greatly appreciated.

which spark version did u download? For e.g. Starting version 2.0, Spark is built with Scala 2.11 by default. Did u download Spark 2.0? — Aravind Yarram, Jan 09 '17 at 20:41
@user155489 Are you using `spark-submit`? Or are you launching class from IDE? — T. Gawęda, Jan 09 '17 at 21:00
https://stackoverflow.com/questions/75947449/run-a-scala-code-jar-appear-nosuchmethoderrorscala-predef-refarrayops — Dmytro Mitin, Apr 07 '23 at 05:04

Aravind Yarram · Answer 1 · 2017-01-09T20:50:43.797

Following is a working pom.xml to create uber jar's for spark applications. Use it by replacing the group and artifact id.

Running mvn clean package builds you the uber (single) jar file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.aravind.spark</groupId>
    <artifactId>wordcount</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <name>${project.artifactId} ${project.version}</name>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <spark.core.version>2.0.2</spark.core.version>
        <spark.sql.version>2.0.2</spark.sql.version>
        <scala.tools.version>2.11</scala.tools.version>
        <scala.version>2.11.8</scala.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.core.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.sql.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <pluginManagement>
            <plugins>
                <plugin>
                    <!-- see http://davidb.github.com/scala-maven-plugin -->
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.1</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                            <configuration>
                                <args>
                                    <arg>-dependencyfile</arg>
                                    <arg>${project.build.directory}/.scala_dependencies</arg>
                                </args>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.19.1</version>
                    <configuration>
                        <useFile>false</useFile>
                        <disableXmlReport>true</disableXmlReport>
                        <!-- If you have classpath issue like NoDefClassError,... -->
                        <!-- useManifestOnlyJar>false</useManifestOnlyJar -->
                        <includes>
                            <include>**/*Test.*</include>
                            <include>**/*Suite.*</include>
                        </includes>
                    </configuration>
                </plugin>
                <plugin>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <version>2.4.1</version>
                    <configuration>
                        <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                        </descriptorRefs>
                    </configuration>
                    <executions>
                        <execution>
                            <id>make-assembly</id>
                            <phase>package</phase>
                            <goals>
                                <goal>single</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <configuration>
                        <source>${maven.compiler.source}</source>
                        <target>${maven.compiler.target}</target>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <!--transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>your main job pipeline class</mainClass>
                                </transformer-->
                            </transformers>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <finalName>${project.artifactId}-${project.version}</finalName>
                            <artifactSet>
                                <excludes>
                                    <exclude>org.scala-lang:scala-library</exclude>
                                    <exclude>org.apache.spark:spark-core_2.10</exclude>
                                    <exclude>log4j:log4j</exclude>
                                    <exclude>org.slf4j:slf4j-api</exclude>
                                    <exclude>ml-apis:xml-apis</exclude>
                                    <exclude>org.apache.httpcomponents:httpclient</exclude>
                                    <exclude>org.apache.httpcomponents:httpcore</exclude>
                                    <exclude>commons-codec:commons-codec</exclude>
                                    <exclude>org.apache.ant:ant</exclude>
                                    <exclude>org.apache.ant:ant-junit</exclude>
                                    <exclude>org.codehaus.jettison</exclude>
                                    <exclude>stax:stax-api</exclude>
                                    <exclude>commons-configuration:commons-configuration</exclude>
                                    <exclude>commons-digester:commons-digester</exclude>
                                    <exclude>commons-beanutils:*</exclude>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

I am getting Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkContext — user155489, Jan 10 '17 at 02:37

score 0 · Answer 2 · answered Jan 12 '17 at 12:04

My entire experience is on Microsoft stack. I just started exploring Java tech. If i asked some thing wrong in the question the person who marked the question down would have helped me and the community by providing explanation why it is marked down..

Any hows I have made my project working by adding the references from project structure > Modules > Dependencies

screen shot

this solved the problem.Thanks all who tried helping me.

scala spark version mismatch

2 Answers2