9

we have a cluster which has about 20 nodes. This cluster is shared among many users and jobs. Therefore, it is very difficult for me to observe my job so that I can get some metrics such as CPU usage, I/O, Network, Memory etc...

How can I get a metrics on job level.

PS: The cluster already have Ganglia installed but not sure how I could get it to work on the job level. What I would like to do is monitor the resource used by the cluster to execute my job only.

diplomaticguru
  • 675
  • 3
  • 8
  • 19
  • Not sure if understand you correctly. You want to get different cluster metrics right from your job? Or you want to know how much resources your job uses? – maxteneff Dec 08 '15 at 13:03
  • @maxteneff, sorry if I was not clear on my post. I want to know how much resources my job used. – diplomaticguru Dec 08 '15 at 13:05
  • I'm curious what you ended up doing to retrieve your metrics? – Saher Ahwal Mar 30 '18 at 17:10
  • Can you also answer the question here: https://stackoverflow.com/questions/49327187/how-to-evaluate-e2e-performance-of-spark-streaming-job-in-particular-the-durat – Saher Ahwal Mar 30 '18 at 17:11

1 Answers1

3

You can get the spark job metrics from Spark History Server, which displays information about:
- A list of scheduler stages and tasks
- A summary of RDD sizes and memory usage
- A Environmental information
- A Information about the running executors

1, Set spark.eventLog.enabled to true before starting the spark application. This configures Spark to log Spark events to persisted storage.
2, Set spark.history.fs.logDirectory, this is the directory that contains application event logs to be loaded by the history server;
3, Start the history server by executing: ./sbin/start-history-server.sh

please refer to below link for more information:
http://spark.apache.org/docs/latest/monitoring.html

Shawn Guo
  • 3,169
  • 3
  • 21
  • 28