1

Compare the following structured queries and their execution plans in web UI.

I use the same dataset for two different queries.

val names = Seq("Jacek").toDF("name")

The following basic query has no SQL metrics in web UI

names.show

enter image description here

Surprisingly the query has number of output rows metric.

names.count

enter image description here

Why?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
  • 1
    @user8371915 Yes. As mentioned on Spark Developers List, LocalRelation does not execute on workers -> there is no event to be send to SparkListener -> No Metrics in Web UI – T. Gawęda Nov 17 '17 at 13:58
  • @T.Gawęda Not really (or at least partially correct) and hence the question. I would not say that _"LocalRelation does not execute on workers"_ as that would not explain why the other query does run on executors and leads to the SQL metric to be displayed. – Jacek Laskowski Nov 17 '17 at 15:03
  • 1
    Jacek, `LocalRelation is a leaf logical plan that allow functions like collect or take to be executed locally, i.e. without using Spark executors.` and Ryan has told, that in that case there is nothing posted to listener – T. Gawęda Nov 17 '17 at 15:15
  • `BroadcastExchangeExec`, `FileSourceScanExec` and `SubqueryExec` are also leaf logical plans and they do "announce" the SQL metrics. See [SQLMetrics.postDriverMetricUpdates](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala#L185-L186). – Jacek Laskowski Nov 17 '17 at 15:33
  • 1
    It's the second part of the question - They announce, but LocalTableScan does not. You can create a PR to invoke postDriverMetricUpdates – T. Gawęda Nov 17 '17 at 15:49

0 Answers0