How is mapPartition different from map?

Question

I am aware of what a mapPartition transformation and map are. Some post claim mapPartition to be faster than map. In what scenarios do we use mapPartition instead of map? Is mapPartition really fast compared to map?

score 1 · Accepted Answer · edited May 23 '17 at 12:25

1

The method map converts each element of the source RDD into a single element of the result RDD by applying a function. mapPartitions converts each partition of the source RDD into multiple elements of the result (possibly none).

It's already answered here: Apache Spark: map vs mapPartitions?

edited May 23 '17 at 12:25

Community

1
1

answered Mar 10 '17 at 19:45

V Sree Harissh

665
5
24

How is mapPartition different from map?

1 Answers1