Spark -Optimizing long running Job

Asked Feb 27 '22 at 07:23

Active Feb 27 '22 at 17:44

Viewed 481 times

we have a spark job that's taking long time to complete, Looked at the spark WebUI and I see lot of shuffling. Couple of things I tried but no luck so far. Increased the sql.shuffle partitions(tried 320,640 and 1600), # of executors (8) and memory (10/12gb) and 4 cores but no significant improvement. Appreciate any guidance on below:

1)when I see the event time line in spark web UI, only one executor is doing most of the processing and rest I don't see any significant activity -

Metrics, lot of difference for shuffle spill between 75th percentile and max -

Any pointers on how to investigate further will be of great help! basically looking for documentation on the event timeline as i see single executor is performing bulk of hte work and how to use the metrics to fix the perf issue by adjusting the spark configuration parameters if thats an option?

edited Feb 27 '22 at 17:44

asked Feb 27 '22 at 07:23

novice8989

Does this answer your question? [Handling Skew data in apache spark production scenario](https://stackoverflow.com/questions/51017314/handling-skew-data-in-apache-spark-production-scenario) – shay__ Feb 27 '22 at 11:57
All the hallmarks of skew. Which version using? – thebluephantom Feb 27 '22 at 12:38
Spark version is :2.4.4.4.1.8.19. Note, its doing a group by and same query if i run in the jupyter notebook, its not showing any slowness. – novice8989 Feb 27 '22 at 17:39
What to infer from the event time line as out of 8 executors only one is doing the work ? – novice8989 Feb 27 '22 at 17:41
Skew that is what we can infer normally – thebluephantom Feb 27 '22 at 20:45
Too little information provided – thebluephantom Feb 27 '22 at 20:48
@thebluephantom , pls let me know what other information I should include in the post? – novice8989 Feb 28 '22 at 02:32
The code...would be a good start... – thebluephantom Feb 28 '22 at 19:59
I tried spark.sql.adaptive.enabled and spark.memory.fraction =0.8 as i read that spark.sql.adaptive.enabled will fix the skew issues if any – novice8989 Feb 28 '22 at 20:40

Spark -Optimizing long running Job

0 Answers0