0

I wanna run a spark job in Intellij IDEA (windows) with spark installed on CentOS 7 (Virtual Machine)

Here is my java code for running Spark installed on Windows:

import org.apache.log4j._
import org.apache.spark._

object RatingsCounter {
  def main(args: Array[String]) {
   Logger.getLogger("org").setLevel(Level.ERROR)

   val sc = new SparkContext("local[*]", "RatingsCounter")

   val lines = sc.textFile("D:\\.....\\......\\abc.data")

   val ratings = lines.map(_.toString.split("\t")(2))

   val results = ratings.countByValue()

   val sortedResults = results.toSeq.sortBy(_._1)

   sortedResults.foreach(println)
  }
}

How can I run the same code in Intellij if Spark is installed on CentOS VM Machine.

PS:

  • Spark 2.4.5
  • Hadoop 2.7

NOTE: I don't wanna use a jar file and spark submit. Have already done that. I wanna run this in Intellij itself and wanna see the result in Intellij only.

Any help is appreciated.

Thanks in advance.

bismi
  • 135
  • 2
  • 12
  • 1
    Local run of spark is always in the same JVM. Since your spark is on a separate machine (even if virtual), it's not in the same JVM. So you can't run it inside IDEA. – RealSkeptic Jun 08 '20 at 17:46
  • 1
    There is no direct support for this in IntelliJ IDEA at the moment, there is a workaround described [in my answer here](https://stackoverflow.com/a/42392922/104891). Does it help? – CrazyCoder Jun 08 '20 at 17:46
  • Thanks @CrazyCoder......unfortunately I don't have an Ultimate Edition of Intellij..... – bismi Jun 08 '20 at 18:29

0 Answers0