How to share SparkContext with methods that need it implicitly

Question

I have the following method:

def loadData(a:String, b:String)(implicit sparkContext: SparkContext) : RDD[Result]

I am trying to test it using this SharedSparkContext: https://github.com/holdenk/spark-testing-base/wiki/SharedSparkContext.

So, I made my test class extend SharedSparkContext:

class Ingest$Test extends FunSuite with SharedSparkContext

And within the test method I made this call:

val res: RDD[Result] = loadData("x", "y")

However, I am getting this error:

Error:(113, 64) could not find implicit value for parameter sparkContext: org.apache.spark.SparkContext
    val result: RDD[Result] = loadData("x", "y")

So how can I make the SparkContext from the testing method visible?

EDIT: I don't see how the question is related with Understanding implicit in Scala

score 0 · Answer 1 · answered Nov 01 '16 at 20:15

What is the variable name of your Spark Context? If it is 'sc' as is typically the case, you will have to alias it to the variable name the method is looking for via implicit val sparkContext = sc and then proceed to call your method in the same environment

How to share SparkContext with methods that need it implicitly

1 Answers1