3

What is the difference between forEachAsync vs forEachPartitionAsync?

If I were to guess here is I would say the following but please correct me if I am wrong.forEachAsync just iterate through values from all partitions one by one in an Async Manner

forEachPartitionAsync: Fan out each partition and run the lambda for each partition in parallel across different workers. The lambda here will Iterate through values from that partition one by one in Async manner

but wait, rdd operations should infact execute in parallel right? so if I call rdd.forEachAsync that should execute in parallel too isn't it? I guess I am a little confused what the difference really is now between forEachAsync vs forEachPartitionAsync? besides passing in Tuple vs Iterator of Tuples to the lambda respectively.

user1870400
  • 6,028
  • 13
  • 54
  • 115

1 Answers1

4

I believe you are already aware of the fact of Async, and asking for the difference between forEach and forEachPartition,

And the difference is, ForEachPartition will allow you to run per partition custom code which you can't do with ForEach.

For Example, You want to save your result to database. Now as you know that opening closing DB connections are costly, one connection(or pool) per executor will be best. So you code would be

rdd.forEachPartition(part => {
    db= mysql..blablabla
    part.forEach(record=> {
    db.save(record)
   })
   db.close()
})

You can't do this in ForEach, in foreach it will iterate for each record.

Remember, One partition will always run on one executor. So if you have any costly pre-work to do before start processing the data use forEachParition. If not just use forEach. Both are parallel. One gives you flexibility other gives simplicity.

RBanerjee
  • 957
  • 1
  • 9
  • 18
  • Thanks much! I get it now. What would you use when retrieving records from a database (say a table scan) ? you get an Iterable and when you do `iterable.iterator.next()` I am not sure if it opens a new connection or uses the existing connection ? but what I can say is that there is no connection initiation or close process given to the user since it is take care by the spark--connector. – user1870400 Apr 03 '17 at 11:08
  • While retrieving data from DB, if you are using Spark JDBC , Spark will internally manage the connection for you. But lets say in a situation where you have some small reference data in DB which you want to pull to do some processing inside forEach, you can use forEachPartition, create your "par partition" connection, pull the data and finally close it when you are done. If accept the answer if it answers you question. Thanks ! – RBanerjee Apr 03 '17 at 11:18