2

May Apache Spark mapPartitions be considered like much performant substitution of Spark UDF with pretty much the same functionality?

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
alexanoid
  • 24,051
  • 54
  • 210
  • 410
  • 1
    UDF is more comparable to `map` rather than `mapPartition`. Please look at https://stackoverflow.com/questions/21185092/apache-spark-map-vs-mappartitions?rq=1 – Sanket9394 Nov 20 '21 at 13:00
  • 2
    `Comparable` - yes, but I ask for the `consideration` while selecting the implementation. In other words - is it a good idea to use `mapPartitions` instead of `UDF` from application performance point of view? – alexanoid Dec 05 '21 at 13:07

0 Answers0