42

Hello I am trying to download spark-core, spark-streaming, twitter4j, and spark-streaming-twitter in the build.sbt file below:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.3",
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

I simply took this libraryDependencies online so I am not sure which versions, etc. to use.

Can someone please explain to me how I should fix this .sbt files. I spent a couple hours trying to figure it out but none of the suggesstion worked. I installed scala through homebrew and I am on version 2.11.8

All of my errors were about: 

Modules were resolved with conflicting cross-version suffixes.
marcospereira
  • 12,045
  • 3
  • 46
  • 52
Bobby
  • 537
  • 1
  • 5
  • 14

2 Answers2

56

The problem is that you are mixing Scala 2.11 and 2.10 artifacts. You have:

scalaVersion := "2.11.8"

And then:

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

Where the 2.10 artifact is being required. You are also mixing Spark versions instead of using a consistent version:

// spark 1.6.1
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"

// spark 1.4.1
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

// spark 0.9.0-incubating
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

Here is a build.sbt that fixes both problems:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)

You also don't need to manually add twitter4j dependencies since they are added transitively by spark-streaming-twitter.

marcospereira
  • 12,045
  • 3
  • 46
  • 52
  • 1
    Thank you so much! One quick question will the version of scala (`2.11.8`) work with all of the spark libraries which have the version `1.6.1`? – Bobby Jun 22 '16 at 04:11
  • 2
    Yes. Spark 1.6.1 has builds for both 2.10 and 2.11 versions of scala. – marcospereira Jun 22 '16 at 04:13
  • I have one more question regarding this. Sorry about that! When I do `sbt run` it seems like the third dependency `"org.apache.spark" %% "spark-core" % sparkVersion` is not being downloaded and therefore in my scala file in `src/main/scala/example.scala` I am getting an error when initializing my spark context: `16/06/21 21:18:14 ERROR SparkContext: Error initializing SparkContext. java.net.UnknownHostException: LM-SFA-11002982: LM-SFA-11002982: nodename nor servname provided, or not known` – Bobby Jun 22 '16 at 04:20
  • 1
    This error is not related to the dependencies being downloaded or not. You should open another question to solve this doubt. – marcospereira Jun 22 '16 at 04:22
  • perfect !!!!!!!!! – abc_spark Mar 13 '22 at 10:46
8

It works for me:

name := "spark_local"

version := "0.1"

scalaVersion := "2.11.8"


libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.5",
  "org.twitter4j" % "twitter4j-stream" % "3.0.5",
  "org.apache.spark" %% "spark-core" % "2.0.0",
  "org.apache.spark" %% "spark-sql" % "2.0.0",
  "org.apache.spark" %% "spark-mllib" % "2.0.0",
  "org.apache.spark" %% "spark-streaming" % "2.0.0"
)
27P
  • 1,183
  • 16
  • 22