0

I want to use ShapeLogic Scala combined with Spark. I am using Scala 2.11.8, Spark 2.1.1 and ShapeLogic Scala 0.9.0. I sucessfully imported the classes to manage images with Spark. Also, I sucessfully compiled and packed (by using SBT) the following application in order to spark-submitting it to a cluster.

The following application simply opens an image and write it to a folder:

// imageTest.scala
import org.apache.spark.sql.SparkSession
import org.shapelogic.sc.io.LoadImage
import org.shapelogic.sc.image.BufferImage
import org.shapelogic.sc.io.BufferedImageConverter

object imageTestObj {
    def main(args: Array[String]) { 
        // Create a Scala Spark Session
        val spark = SparkSession.builder().appName("imageTest").master("local").getOrCreate();

        val inPathStr = "/home/vitrion/IdeaProjects/imageTest";
        val outPathStr = "/home/vitrion/IdeaProjects/imageTest/output";
        val empty = new BufferImage[Byte](0, 0, 0, Array());
        var a = Array.fill(3)(empty);
        for (i <- 0 to 3) {
            val imagePath = inPathStr + "IMG_" + "%01d".format(i + 1);
            a(i) = LoadImage.loadBufferImage(inPathStr);
        }
        val sc = spark.sparkContext;
        val imgRDD = sc.parallelize(a);
        imgRDD.map { outBufferImage =>
            val imageOpt = BufferedImageConverter.bufferImage2AwtBufferedImage(outBufferImage)
            imageOpt match {
                case Some(bufferedImage) => {
                    LoadImage.saveAWTBufferedImage(bufferedImage, "png", outPathStr)
                    println("Saved " + outPathStr)
                }
                case None => {
                    println("Could not convert image")
                }
            }
        }
    } 
}

This is my SBT file

name := "imageTest"

version := "0.1"

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.1.1" % "provided",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" % "provided",
  "org.shapelogicscala" %% "shapelogic" % "0.9.0" % "provided"
)

However, the following error appears. When the package SBT command is executed, it seems like the ShapeLogic Scala dependencies are not included in the application JAR:

[vitrion@mstr scala-2.11]$ pwd
/home/vitrion/IdeaProjects/imageTest/target/scala-2.11
[vitrion@mstr scala-2.11]$ ls
classes  imagetest_2.11-0.1.jar  resolution-cache
[vitrion@mstr scala-2.11]$ spark-submit --class imageTestObj imagetest_2.11-0.1.jar 
Exception in thread "main" java.lang.NoClassDefFoundError: org/shapelogic/sc/image/BufferImage
    at imageTestObj.main(imageTest.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.shapelogic.sc.image.BufferImage
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 10 more

I hope someone can help me to solve it? Thank you very much

Vitrion
  • 405
  • 5
  • 14

2 Answers2

2

This error says everything:

Caused by: java.lang.ClassNotFoundException: org.shapelogic.sc.image.BufferImage
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

Add this missing dependency (ShapeLogic class) ------> org.shapelogic.sc.image.BufferImage, that should resolve the issue. Maven or SBT both should give the same error, if you miss this dependency!!

Since you are working on the cluster mode, you can directly add dependencies using --jars on spark-submit, please follow this post for more details.

These threads might help you: Link1 Link2

  • Thank you very much for your response. I understand that the BufferImage class is not included in the JAR file when packaging the application. But, how can I include it? As I said before, SBT compiles and packages sucessfully – Vitrion Aug 23 '17 at 17:51
  • @Vitrion are you working on cluster mode or client mode? See this https://stackoverflow.com/questions/34272426/how-to-give-dependent-jars-to-spark-submit-in-cluster-mode?rq=1 might help you –  Aug 23 '17 at 20:15
  • Yes I am working in cluster mode. Everything has sense. I already submitted the application with an additional JAR file (ShapeLogic Scala JAR) and everything is working again. Thank you very much! – Vitrion Aug 24 '17 at 00:25
  • Would be great if you modify your original response, so anyone that faces this problem can solve it also. So, I will mark it as the solution. – Vitrion Aug 24 '17 at 00:28
  • I have updated the answer, don't forget to accept the answer. –  Aug 24 '17 at 00:34
0

Your dependencies listed in sbt files will not be included by default in your jar submitted to spark, so for sbt you have to use a plugin to build an uber/fat jar that would include shapelogicscala classes. You can use this link on SO, How to build an Uber JAR (Fat JAR) using SBT within IntelliJ IDEA? to see how you can manage this with sbt.

dumitru
  • 2,068
  • 14
  • 23