I am working with a Spark/Scala application on intelliJ with Maven. I am trying to write a file and after the writing is complete, I am trying to read it line by line and increment a variable named count depending on a condition. Here is my code:
val writer: BufferedWriter = new BufferedWriter(new FileWriter("test.csv"))
/* Logic
for
writing
the file
*/
writer.close()
var count = 0
val bufferedSource = Source.fromFile("test.csv")
for(line <- bufferedSource.getLines()){
count = count+1
line.split(",").foreach(x => {if(x.toString == "1") count = count+1})
}
bufferedSource.close()
When I run the above code, it gives me the following error:
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.IntRef.create(I)Lscala/runtime/IntRef;
It is throwing the error at the variable declaration statement: var count = 0
I searched for this error online and found the reason to be scala version mismatch, as pointed out here: NoSuchMethodError when declaring a variable
Initially, I had my project SDK setup to 2.10.6. That's the default version I use for my projects. I didn't mention any scala language dependency in my pom.xml. And when I run the mvn -U package command for building the jar, I get the following messages:
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building project
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ IPscan ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\zkh9xcl\workspace\IPscan\src\main\resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ IPscan ---
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.databricks:spark-csv_2.11:1.4.0 requires scala version: 2.11.7
[WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ IPscan ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ IPscan ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\zkh9xcl\workspace\IPscan\src\test\resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile) @ IPscan ---
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.databricks:spark-csv_2.11:1.4.0 requires scala version: 2.11.7
[WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ IPscan ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ IPscan ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ IPscan ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ IPscan ---
[INFO] Including com.databricks:spark-csv_2.11:jar:1.4.0 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.11.7 in the shaded jar.
[INFO] Including org.apache.commons:commons-csv:jar:1.1 in the shaded jar.
[INFO] Including com.univocity:univocity-parsers:jar:1.5.1 in the shaded jar.
[INFO] Skipping pom dependency org.jodd:jodd:pom:3.4.0 in the shaded jar.
[INFO] Including org.threeten:threetenbp:jar:1.3.3 in the shaded jar.
[INFO] Attaching shaded artifact.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.054 s
[INFO] Finished at: 2017-10-26T11:18:39-04:00
[INFO] Final Memory: 38M/663M
[INFO] ------------------------------------------------------------------------
It is throwing a warning saying: [WARNING] Multiple versions of scala libraries detected!
Here's my pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>project</groupId>
<artifactId></artifactId>
<version>1.0-SNAPSHOT</version>
<name>IPscan</name>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.apache.http</pattern>
<shadedPattern>org.shaded.apache.http</shadedPattern>
</relocation>
</relocations>
<filters>
<filter>
<artifact>*:*</artifact>
</filter>
</filters>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>shaded</shadedClassifierName>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.11</artifactId>
<version>1.4.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.jodd/jodd -->
<dependency>
<groupId>org.jodd</groupId>
<artifactId>jodd</artifactId>
<version>3.4.0</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.0</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.threeten/threetenbp -->
<dependency>
<groupId>org.threeten</groupId>
<artifactId>threetenbp</artifactId>
<version>1.3.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>1.6.0</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
I am new to maven, please excuse me if there is a probelm with my pom.xml.
I am not sure why there are multiple versions of scala libraries included in the jar. I actually changed the java version to 2.11.7 and then to 2.10.4 in the global libraries of the project in intelliJ, but it gives a problem saying Library scala-sdk-2.10.4 is not used [Fix]. I am pretty sure I added 2.10.4 to the list of dependencies in intelliJ. No matter what I do, I still get the multiple scala versions warning when I build the jar and the noSuchMethodError still persists. Looks like I am missing something.
What is causing the error? And what is the best way to overcome this and avoid it in the future? Any help would be appreciated. Thank you!
P.S: I am using Spark 1.6 and I can't upgrade to a newer version.
