Spark testing base strange validation failure

Question

When I execute sbt test in order to test a spark application utilizing spark-testing-base

Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0)) did not equal Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0))

for a test case of

val input: Map[String, Any] = Map("digits" -> Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 0))
val expectedOutput: Map[String, Any] = Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0))
val result = SimpleContext.runTheJOb(session, input)

Minimal example can be found https://github.com/geoHeil/apache-spark-restAPI-example

edit

the whole test case directly

class SimpleTest extends FunSuite with SharedSparkContext with DatasetSuiteBase {

  test("SimpleContext should multiply input numbers by 3") {
    val session = spark

    val input: Map[String, Any] = Map("digits" -> Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 0))
    val expectedOutput: Map[String, Any] = Map("result" -> Array(3, 6, 9, 12, 15, 18, 21, 24, 27, 0))
    val result = SimpleContext.runTheJOb(session, input)
    assert(expectedOutput === result)
  }
}

@LostInOverflow the minimal "code" is already included. But if you prefer to have the whole code I will edit the question. — Georg Heiler, Nov 30 '16 at 20:43
Thanks! Your gists can become invalid at any point and this questions should serve not only you. — , Nov 30 '16 at 20:47

score 4 · Accepted Answer · edited May 23 '17 at 12:01

4

The problem you are having is that Scala Arrays, being Java arrays under the hood, are not comparable using == (see this answer for an explanation).

Example:

scala> Array(1,2) == Array(1,2)
res0: Boolean = false

However most of the other collections are comparable:

scala> List(1,2) == List(1,2)
res1: Boolean = true

Your options would be to either use one of the other collections (like a List) or to use deep for comparisons:

scala> Array(1,2).deep == Array(1,2).deep
res22: Boolean = true

scala> Array(1,2).deep == Array(1,3).deep
res23: Boolean = false

edited May 23 '17 at 12:01

Community

1
1

answered Nov 30 '16 at 20:42

evan.oman

5,922
22
43

2

so Array(1, 2) should equal (Array(1, 2)) via http://www.scalatest.org/user_guide/using_matchers#checkingEqualityWithMatchers should be the solution. – Georg Heiler Nov 30 '16 at 20:45

Spark testing base strange validation failure

edit

1 Answers1