54

I'am trying to build spark streaming application using sbt package,I can't discover what's the reason of this error.

this is some thing of the error

scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found. at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16) at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17) at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48) at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40) at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)

and here is the code

import org.apache.spark.SparkContext
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.twitter._
import twitter4j.Status
object TrendingHashTags {
def main(args: Array[String]): Unit = {
val Array(consumerKey, consumerSecret, accessToken, accessTokenSecret,
lang, batchInterval, minThreshold, showCount ) = args.take(8)
val filters = args.takeRight(args.length - 8)
System.setProperty("twitter4j.oauth.consumerKey", consumerKey)
System.setProperty("twitter4j.oauth.consumerSecret", consumerSecret)
System.setProperty("twitter4j.oauth.accessToken", accessToken)
System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret)
val conf = new SparkConf().setAppName("TrendingHashTags")
val ssc = new StreamingContext(conf, Seconds(batchInterval.toInt))
val tweets = TwitterUtils.createStream(ssc, None, filters)
val tweetsFilteredByLang = tweets.filter{tweet => tweet.getLang() == lang}
val statuses = tweetsFilteredByLang.map{ tweet => tweet.getText()}
val words = statuses.flatMap{status => status.split("""\s+""")}
val hashTags = words.filter{word => word.startsWith("#")}
val hashTagPairs = hashTags.map{hashtag => (hashtag, 1)}
val tagsWithCounts = hashTagPairs.updateStateByKey(
(counts: Seq[Int], prevCount: Option[Int]) =>
prevCount.map{c => c + counts.sum}.orElse{Some(counts.sum)}
)
val topHashTags = tagsWithCounts.filter{ case(t, c) =>
c > minThreshold.toInt
}
val sortedTopHashTags = topHashTags.transform{ rdd =>
rdd.sortBy({case(w, c) => c}, false)
}
sortedTopHashTags.print(showCount.toInt)
ssc.start()
ssc.awaitTermination()
}
}
Community
  • 1
  • 1
Elsayed
  • 2,712
  • 7
  • 28
  • 41

4 Answers4

78

I solved this issue ,I found that I used java 9 that isn't compatible with scala version so I migrated from java 9 into java 8.

Elsayed
  • 2,712
  • 7
  • 28
  • 41
  • 5
    I wish I found this answer before trying to fix my pom.xml for two days. Thanks mate! – Hubert Grzeskowiak Sep 11 '18 at 03:14
  • 1
    Keywords for future Googlers: Maven Scala Java JDK Gatling – Hubert Grzeskowiak Sep 11 '18 at 03:16
  • 1
    Link for changing default JDK: https://stackoverflow.com/questions/21964709/how-to-set-or-change-the-default-java-jdk-version-on-os-x – kevin_theinfinityfund Jan 19 '20 at 22:51
  • Tip: ```sudo update-alternatives --config java[Tab]``` ```sudo update-alternatives --config javac[Tab]``` Check if ~/.bashrc file has a hard-coded value for ```$JAVAHOME```, maven reads $JAVAHOME to decide which version of java should be used. So, ensure that ```$JAVAHOME``` value also reflects the change effected in java version. – sherminator35 Nov 09 '20 at 16:58
5

The error means that scala was compiled using a version of java, different from the current version.

I am using maven instead of sbt, but the same behavior is observed.

Find the java version:

> /usr/libexec/java_home -V
Matching Java Virtual Machines (2):
    15.0.1, x86_64:     "OpenJDK 15.0.1"        /Users/noname/Library/Java/JavaVirtualMachines/openjdk-15.0.1/Contents/Home
    1.8.0_271, x86_64:  "Java SE 8"     /Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home

If you installed scala, while you were on version >1.8 and then downgraded the java version (edited the $JAVA_HOME to point to 1.8), you will get this error.

Checked the scala version being used by the project :

$ ls -l /Users/noname/.m2/repository/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.jar
-rwxrwxrwx  1 noname  staff  0 Nov 17 03:41 /Users/noname/.m2/repository/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.jar

To rectify the issue, remove the scala jar file:

$ rm /Users/noname/.m2/repository/org/scala-lang/scala-library/2.11.11/scala-library-2.11.11.jar

Now, execute mvn clean install again and the project would compile.

ForeverLearner
  • 1,901
  • 2
  • 28
  • 51
2

I had faced this issue when I had to downgrade my projects Scala version to use a dependency that was compiled in a lower Scala version and could not resolved it even after I made sure JDK and all other dependencies are compatible with the downgraded Scala library version.

As @ForeverLearner mentioned above, deleting Scala library versions higher than the one I am now using to compile project from maven repo (/Users/<>/.m2/repository/org/scala-lang/scala-library/...) helped me get rid of this error

grover
  • 21
  • 1
0

The above fix resolved my issue as well (setting Java 8) , If you are using Intellij you can go to Project Settings and under Project change the Project SDK to 1.8 .

RaHuL VeNuGoPaL
  • 447
  • 5
  • 7