1

I'm trying to create a new Spark application based on Hortonworks tutorial.But the build.sbt keeps throwing the below error when i try to import the necessary modules.

I have SCALA - 2.11.4 AND SBT 0.13 installed.Any help ??

Error:Error while importing SBT project:<br/>...<br/><pre>[warn]    ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]      org.scala-lang:scala-library:2.11.4 ((sbt.Classpaths) Defaults.scala#L1340)
[warn]        +- sparktutorialscala:sparktutorialscala_2.11:1.0
[warn]      org.apache.spark:spark-core_2.11:2.1.0 (C:\Users\USERNAME\IdeaProjects\HelloWorld\build.sbt#L7-13)
[warn]        +- sparktutorialscala:sparktutorialscala_2.11:1.0
[warn]      org.scala-lang:scala-compiler:2.11.4
[warn]        +- sparktutorialscala:sparktutorialscala_2.11:1.0
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.scala-lang#scala-library;2.11.4: not found
[error] unresolved dependency: org.apache.spark#spark-core_2.11;2.1.0: not found
[error] unresolved dependency: org.scala-lang#scala-compiler;2.11.4: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: org.scala-lang#scala-library;2.11.4: not found
[error] unresolved dependency: org.apache.spark#spark-core_2.11;2.1.0: not found
[error] unresolved dependency: org.scala-lang#scala-compiler;2.11.4: not found
[error] Total time: 66 s, completed 17-Oct-2017 10:49:23
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true</pre><br/>See complete

build.sbt

name := "sparkTutorialScala"

version := "1.0"

scalaVersion := "2.11.4"

libraryDependencies ++= {
  val sparkVer = "2.1.0"
  Seq(
    "org.apache.spark" %% "spark-core" % sparkVer % "provided" withSources()
  )
}

Updated Error message :

  [error] unresolved dependency: org.apache.spark#spark-core_2.11;2.2.0: public: unable to get resource for org/apache/spark#spark-core_2.11;2.2.0: res=https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.2.0/spark-core_2.11-2.2.0.pom: java.net.ConnectException: Failed to connect to repo1.maven.org/151.101.60.209:443
[error] unresolved dependency: org.scala-lang#scala-compiler;2.11.8: public: unable to get resource for org/scala-lang#scala-compiler;2.11.8: res=https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.8/scala-compiler-2.11.8.pom: java.net.ConnectException: Failed to connect to repo1.maven.org/151.101.60.209:443
[error] (*:ssExtractDependencies) sbt.librarymanagement.ResolveException: unresolved dependency: org.scala-lang#scala-library;2.11.8: public: unable to get resource for org/scala-lang#scala-library;2.11.8: res=https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.pom: java.net.ConnectException: Failed to connect to repo1.maven.org/151.101.60.209:443
[error] unresolved dependency: org.apache.spark#spark-core_2.11;2.2.0: public: unable to get resource for org/apache/spark#spark-core_2.11;2.2.0: res=https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.2.0/spark-core_2.11-2.2.0.pom: java.net.ConnectException: Failed to connect to repo1.maven.org/151.101.60.209:443
[error] unresolved dependency: org.scala-lang#scala-compiler;2.11.8: public: unable to get resource for org/scala-lang#scala-compiler;2.11.8: res=https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.8/scala-compiler-2.11.8.pom: java.net.ConnectException: Failed to connect to repo1.maven.org/151.101.60.209:443
[error] Total time: 9 s, completed 19-Oct-2017 15:11:47
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true</pre><br/>See complete log in <a href="file:/C:/Users/
theDbGuy
  • 903
  • 1
  • 9
  • 22
  • 1
    Show your sbt file please – Yehor Krivokon Oct 17 '17 at 09:57
  • @yehor Pls check my edited post for the SBT – theDbGuy Oct 17 '17 at 09:58
  • You sure you're showing us all you have in your project? See the error: `unresolved dependency: org.apache.spark#spark-core_2.11_2.11;2.1.0: not found` where `_2.11` is repeated twice (!) – Jacek Laskowski Oct 17 '17 at 20:38
  • Can you please change `scalaVersion := "2.11.4"` to use `2.11.8` and `val sparkVer = "2.1.0"` to use `2.2.0`? Are you behind a proxy? Please install the very recent sbt 1.0.2. – Jacek Laskowski Oct 17 '17 at 20:40
  • @Jacek ,i did the same as mentioned but its still failing.Yes, I'm behind a proxy and i have setup the proxy in IntelliJ appropriately. – theDbGuy Oct 19 '17 at 13:51
  • Has it got anything to do with the this 'http://repo1.maven.org/151.101.60.209:443' link? Bcoz,I tried accessing it manually but i could not find the mentioned link. – theDbGuy Oct 19 '17 at 14:15
  • @theDbGuy Could https://stackoverflow.com/a/18824439/1305344 be of help perhaps? – Jacek Laskowski Oct 19 '17 at 22:55
  • Adding to this,my firewall has some restrictions which was blocking IntelliJ to communicate. Turning off the firewall condition made this work. – theDbGuy Feb 19 '18 at 05:19

1 Answers1

0

Well,SBT 1.0 was not compatible with the version of IDEAJ plugin or spark that I was using.Installing SBT 0.13 instead of the latest one resolved the issue.

theDbGuy
  • 903
  • 1
  • 9
  • 22