Here is my code example:
case class Person(name:String,tel:String){
def equals(that:Person):Boolean = that.name == this.name && this.tel == that.tel}
val persons = Array(Person("peter","139"),Person("peter","139"),Person("john","111"))
sc.parallelize(persons).distinct.collect
It returns
res34: Array[Person] = Array(Person(john,111), Person(peter,139), Person(peter,139))
Why distinct doesn't work?I want the result as Person("john",111),Person("peter",139)
Extending further from the observation of @aaronman, there is a workaround for this issue.
On the RDD, there're two definitions for distinct:
/**
* Return a new RDD containing the distinct elements in this RDD.
*/
def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] =
map(x => (x, null)).reduceByKey((x, y) => x, numPartitions).map(_._1)
/**
* Return a new RDD containing the distinct elements in this RDD.
*/
def distinct(): RDD[T] = distinct(partitions.size)
It's apparent from the signature of the first distinct that there must be an implicit ordering of the elements and it's assumed null if absent, which is what the short version .distinct() does.
There's no default implicit ordering for case classes, but it's easy to implement one:
case class Person(name:String,tel:String) extends Ordered[Person] {
def compare(that: Person): Int = this.name compare that.name
}
Now, trying the same example delivers the expected results (note that I'm comparing names):
val ps5 = Array(Person("peter","138"),Person("peter","55"),Person("john","138"))
sc.parallelize(ps5).distinct.collect
res: Array[P5] = Array(P5(john,111), P5(peter,139))
Note that case classes already implement equals and hashCode, so the impl on the provided example is unnecessary and also incorrect. The correct signature for equals is: equals(arg0: Any): Boolean -- BTW, I first thought that the issue had to do with the incorrect equals signature, which sent me looking in the wrong path.
For me the problem was related to object equality, as mentioned by Martin Odersky in Programming in Scala (chapter 30), although I have a normal class (not a case class). For a correct equality test, you must re-define (override) hashCode() if you have a custom equals(). Also you need to have a canEqual() method for 100% correctness. I haven't looked at the implementation details of an RDD, but since it is a collection, probably it uses some complex/parallel variation of a HashSet or other hash-based data structure for comparing objects and ensuring distinctness.
Declaring hashSet(), equals(), canEqual(), and compare() methods solved my problem:
override def hashCode(): Int = {
41 * (41 + name.hashCode) + tel.hashCode
}
override def equals(other: Any) = other match {
case other: Person =>
(other canEqual this) &&
(this.name == other.name) && (this.tel == other.tel)
case _ =>
false
}
def canEqual(other: Any) = other.isInstanceOf[Person]
def compare(that: Person): Int = {
this.name compare that.name
}
As others have pointed out this is a bug in spark 1.0.0. My theory as to where it is coming from is that if you look at the diff of 1.0.0 to 9.0 you see
- def repartition(numPartitions: Int): RDD[T] = {
+ def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = {
And if you run
case class A(i:Int)
implicitly[Ordering[A]]
You get an error
<console>:13: error: No implicit Ordering defined for A.
implicitly[Ordering[A]]
So I think the workaround is define an implicit ordering for a the case class, unfortunately I'm not a scala expert but this answer seems to do it correctly
