3

I have a local hadoop single node and hive installed and I have some hive tables stored in hdfs. Then I configure Hive with MySQL Metastore. And now I installed spark and Im doing some queries over hive tables like this (in scala):

var hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
result = hiveContext.sql("SELECT * FROM USERS");
result.show

Do you know how to configure spark to show to the execution time of the query? Because for default it is not showing..

codin
  • 743
  • 5
  • 15
  • 27
  • Possible duplicate of this http://stackoverflow.com/questions/34629313/how-to-measure-the-execution-time-of-a-query-on-spark – charles gomes Apr 03 '16 at 18:36
  • OMG, isn't there anyone who finally doesn't suggest to visit a web browser to get the execution time?! can't believe it. This has to also be possible **programmatically**. – Nadjib Mami Oct 19 '16 at 08:03

1 Answers1

2

Use spark.time().

var hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) result = hiveContext.sql("SELECT * FROM USERS"); spark.time(result.show)

https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting

Tyrone321
  • 1,702
  • 15
  • 23