0

I have a spark application that involves 2 scala companion objects as follows. 

object actualWorker {
  daoClient

  def update (data, sc) {
    groupedData = sc.getRdd(data).filter. <several_operations>.groupByKey
    groupedData.foreach(x => daoClient.load(x))
  }
}


object SparkDriver {
  getArgs
  sc = getSparkContext
  actualWorker.update(data, sc : sparkContext)

}

The challenge I have is in writing unit-test for this spark application. I am using Mockito and ScalaTest, Junit for these tests.

I am not able to mock the daoClient while writing the unit test. [EDIT1: Additional challenge is the fact that my daoClient is not serializable. Because I am running it on spark, I simply put it in an object (not class) and it works on spark; but it makes it non unit-testable ]

I have tried the following:

  1. Make ActualWorker a class that can have a uploadClient passed in the Constructor. Create a client and instantiate it in Actual Worker Problem: Task not serializable exception.
  2. Introduce a trait for upload client. But still I need to instantiate a client at some point in the SparkDriver, which I fear will cause the Task Not serializable exception.

Any inputs here will be appreciated.

PS: I am fairly new to Scala and spark

TheGT
  • 412
  • 8
  • 13
  • You cannot mock an `object` - it makes no sense. You can mock a class, but then the class must extend Serializable (and all class variables must be serializable as well) – Glennie Helles Sindholt Sep 29 '15 at 10:12
  • I have updated my question about why I used object over class for actualWorker. see EDIT1. – TheGT Sep 29 '15 at 14:41
  • I am beginning to find http://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou very helpful. I will update once I have a working solution. Meanwhile leaving the question open until anyone has a better/nicer solution. – TheGT Sep 29 '15 at 15:03

1 Answers1

0

While technically not exactly a unit testing framework, I've used https://github.com/holdenk/spark-testing-base to test my Spark code and it works well.

Jeff L
  • 141
  • 1
  • 1
  • 7