More information can be found in the official documentation.
Questions tagged [spark-webui]
12 questions
4
votes
1 answer
Explanation about Executor Summary in Spark Web UI
I have a few questions about the Executor section of Spark Web UI:
I see two numbers such as 0.0 B / 434.4 MiB under Storage Memory, On Heap Storage Memory, and Off Heap Storage Memory, what are those?
Also is Storage Memory the sum of On Heap…
kee
- 10,969
- 24
- 107
- 168
2
votes
0 answers
Why executor memory used is shown greater than total available memory on spark web UI?
I have a spark structured streaming job that is running for around last 3 weeks. When I open the Executors tab on spark web UI, it shows
memory used - 36.1GB
total available memory for storage - 3.2GB
For this application executor memory is set…
Ravi
- 23
- 7
2
votes
1 answer
Why does metric "number of output rows" in Apache Spark UI shows a value higher than the size of the table when the table is used multiple times?
I've first encountered this behavior in query 47 of TPCDS benchmark.
For clarification this is the query.
--q47.sql-- …
Panos
- 557
- 5
- 13
1
vote
1 answer
Is it possible to combine jobs in the Spark UI?
We are running an optimization routine in Spark that makes many (e.g. 100s) of gradient calls. Each gradient evaluation is listed as its own Spark job in the UI. Is it possible to somehow collapse all these calls into a single job?
David
- 213
- 1
- 8
1
vote
0 answers
Understanding Apache Spark Web UI performance metrics
I'm new to Spark and I'm trying to understand the metrics in the Web UI that are related to in my Spark Application (developed through Dataset API). I've watched few videos by Spark Summit and Databricks and most of the videos I watched were about a…
1
vote
2 answers
apache spark: Completed application history deleted after restarting
When I restart spark cluster all of history of completed application in web ui are deleted. How can I preserve this history from deleting when restarting?
DAVID_ROA
- 309
- 1
- 3
- 18
1
vote
1 answer
How to get web UI information like "Summary Metrics for Completed Tasks" in Stages tab programmatically?
I want to get "Summary Metrics for Completed Tasks" in my scala code. This information can be found in the Spark webUI:
fan
- 11
- 3
1
vote
0 answers
When would physical operator (e.g. LocalTableScanExec) show no SQL metrics in web UI?
Compare the following structured queries and their execution plans in web UI.
I use the same dataset for two different queries.
val names = Seq("Jacek").toDF("name")
The following basic query has no SQL metrics in web UI
names.show
Surprisingly…
Jacek Laskowski
- 72,696
- 27
- 242
- 420
0
votes
1 answer
Spark History server setup
I am trying to setup Spark History config server in local. I am using using Windows and Pycharm for Pyspark programming. I am able to view Spark Web-UI at localhost:4040.
The things I have done are:
spark-defaults.conf: (Where I have added last…
0
votes
2 answers
Can't connect to spark web UI inside open shift container
I'm running my spark application in open shift container. The application runs for almost 2-4 hours. I do get the message of sparkUI started at http://hostname:4040. But when I click on it ,I'm getting webpage not found even though the application…
Aditya
- 11
- 4
0
votes
1 answer
Hide Spark environment variable value from ps and web-ui
I am using Spark 2.3.1 on Mac, in Java.
I have confidential security info stored in an environment variable. However, as it's confidential, I don't want to expose its value through ps -e nor from http://localhost:4040/environment/.
Is there a way…
sue
- 43
- 4
-1
votes
2 answers
Spark-shell: Web UI doesn't change when I execute process
I use Spark in local mode. I run spark-shell and use a file as a data set. All work very good (for example, I ask spark-shell to count the number of words which begin by "a" in the file and I have the good result), but when I see at the web UI, it…
Fitz
- 41
- 4