1

I am building a Spark application with bash script and I have an only spark-sql and core dependencies in the build.sbt file. So every time I call some rdd methods or convert the data to case class for dataset creation I get this error:

Caused by: java.lang.NoClassDefFoundError: scala/Product$class

I suspect that it is a dependency error. So how should I change my dependencies to fix this?

dependencies list:

import sbt._

object Dependencies {
  lazy val scalaCsv = "com.github.tototoshi" %% "scala-csv" % "1.3.5"
  lazy val sparkSql = "org.apache.spark" %% "spark-sql" % "2.3.3"
  lazy val sparkCore = "org.apache.spark" %% "spark-core" % "2.3.3"
}

build.sbt file:

import Dependencies._
lazy val root = (project in file(".")).
  settings(
    inThisBuild(List(
      scalaVersion := "2.11.12",
      version      := "test"
    )),
    name := "project",
    libraryDependencies ++= Seq(scalaCsv, sparkSql, sparkCore),
    mainClass in (Compile, run) := Some("testproject.spark.Main")
  )

I launch spark app with spark 2.3.3 as my spark home directory like this:

#!/bin/sh
$SPARK_HOME/bin/spark-submit \
  --class "testproject.spark.Main " \
  --master local[*] \
  target/scala-2.11/test.jar
Cassie
  • 2,941
  • 8
  • 44
  • 92

1 Answers1

0

Not sure what was the problem exactly, however, I have recreated the project and moved the source code there. The error disappeared

Cassie
  • 2,941
  • 8
  • 44
  • 92