0

I am trying to run Performance testing on one of my spark jobs which loads data into memory and then perform spark-sql operations on the data and finally returns the result to user.

I would like to know how much time does the job takes in each stage: i.e. time taken by the spark job to load data into memory and time taken by the job to complete its execution. Is there any good profiling tool available for this.

If not, Can I use already available spark UI/logs to fetch such information.

Or, if there is any other way to get this data.

Please note that I am a newbie in Spark so please excuse for any illogical/irrelevant question.

Thanks in advance!

Amandeep Singh
  • 297
  • 4
  • 19

1 Answers1

1

The application UI (default port 4040 on the master) has the "Stages" tab, which reports the duration of each stage. It will look something like this: like this:

akn320
  • 573
  • 4
  • 13