shuffle last task taking too much time to complete

Question

enter image description hereI have around 80GB of data , everything is going smooth till last shuffle task comes up ,all the task are getting finished within 30 mins, but last task takes more than 2 hours to complete it . enter image description here Joins : (left join) Joining 3-tables , one of the table is small relatively (2 MB )data , for that setting broadcast variable , even I removed that 3rd table , It did not resolved my issue .

below is the parameters that configured .

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "904857600")
spark.conf.set("spark.cleaner.referenceTracking.blocking", "false")
spark.conf.set("spark.cleaner.periodicGC.interval", "5min")
spark.conf.set("spark.default.parallelism","6000")
spark.conf.set("spark.sql.shuffle.partitions","2000")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

Are any of the tasks in the job taking a lot longer than the others? — Matt Andruff, Mar 02 '22 at 14:25
Have you looked at the query plan to see what is happening in that shuffle? Can you filter/sort/bucket the data any further to reduce the workload? — Matt Andruff, Mar 02 '22 at 14:27
Task or job ? They're two different things, your answer does not demonstrate you understand the difference — Matt Andruff, Mar 02 '22 at 14:28
There are 2000 task , in out of 1999 has been executed successfully , except last one . Just wanted to know like , when it happened , and if possible provide me some solution. — Sanjiv Kumar, Mar 02 '22 at 14:39
Can you add us the DAG view? Because it might show what is attempting your job at it's very end, that takes so much time. A last transfer whose keys don't matches? — Marc Le Bihan, Mar 02 '22 at 14:42

score 0 · Accepted Answer · answered Mar 02 '22 at 14:52

You are suffering from data Skew. Essentially most of the work is being done by 1 node instead of the work being distributed across multiple nodes. This is why I wanted clarification of job or [task/stage].

You should consider adding a salt to your join key to help distribute the work across multiple nodes. It will require more shuffles but it will lesson the impact on one node doing all the work.

Add salt to all columns in the join
Do your 3 way Join with salt column included.
Then do a secondary group by to remove the salt from the query.

This will better distribute the work.

I used AQE to solve it. – Sanjiv Kumar May 06 '22 at 08:53 — Sanjiv Kumar, May 06 '22 at 08:53

shuffle last task taking too much time to complete

1 Answers1