0

This is an SBT build file which I have written

name := "HelloSpark"

version := "1.0"

scalaVersion := "2.12.2"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"

when I run sbt package I get error message

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11;2.1.1: not found

[error] Total time: 2 s, completed Oct 29, 2017 1:01:13 AM

Community
  • 1
  • 1
  • In general, you can't use one version of the _Scala_ compiler with a library built with a different version. Here you are specifying a _Scala_ 2.12 compiler with a library built for _Scala_ 2.11. That's not going to work. To ensure libraries match the compiler, you can skip the version suffix if you use the`%%` operator between the organization and artifact name. That is, use a dependency of `"org.apache.spark" %% "spark-core" % "2.1.1"` instead of `"org.apache.spark" % "spark-core_2.11" % "2.1.1"`. _Spark_ doesn't currently support 2.12, so set `scalaVersion := "2.11.11"`. – Mike Allen Oct 29 '17 at 03:33
  • Are you using a corporate network? Are you perhaps behind a proxy? Can you show the entire log from `sbt package`? (edit your question and paste the output). Thanks. – Jacek Laskowski Oct 29 '17 at 16:18

1 Answers1

2

Use Scala 2.11

scalaVersion := "2.11.11"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

using scala 2.12 with spark 2.1

Spark does not support Scala 2.12

Just in case, you can find binaries for spark-core here: https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
  • HI Dmytro, Thanks for ypur suggestion. It makes sense now. However I have scala 2.12 installed in my system. and changing the scala version is throwing another error. --- at java.lang.Thread.run(Thread.java:748) [error] (*:update) sbt.ResolveException: unresolved dependency: org.scala-lang#scala-library;2.11.11: not found [error] unresolved dependency: org.scala-lang#scala-compiler;2.11.11: not found [error] Total time: 4 s, completed Oct 29, 2017 3:56:44 AM . -- Could you please suggest any workaround –  Oct 28 '17 at 22:28
  • @devD What is your version of sbt? – Dmytro Mitin Oct 29 '17 at 00:04
  • My version of sbt is 0.13.16 –  Oct 29 '17 at 01:12
  • 2
    @devD It doesn't matter what version of _Scala_ is installed on your system, as _sbt_ will use the version specified by `scalaVersion`. It looks to me like you have a network problem. Do you access the Internet through a proxy? – Mike Allen Oct 29 '17 at 03:35