Where is the Spark UI on Google Dataproc?

Question

What port should I use to access the Spark UI on Google Dataproc?

I tried port 4040 and 7077 as well as a bunch of other ports I found using netstat -pln

Firewall is properly configured.

score 26 · Accepted Answer · answered Oct 18 '15 at 01:14

Dataproc runs Spark on top of YARN, so you won't find the typical "Spark standalone" ports; instead, when running a Spark job, you can visit port 8088 which will show you the YARN ResourceManager's main page. Any running Spark jobs will be accessible through the Application Master link on that page. The Spark Application Master's page looks the same as the familiar Spark-standalone landing page that you would normally find on port 8080 for default Spark setups.

Since workers check in over the internal network, YARN's links will be using cluster-internal hostnames (the hostnames should include your Dataproc cluster name as a prefix), but this means if you're accessing from the outside network, the links may not work at first; you have to replace the hostname with the external IP address if you're using the firewall-based approach.

An easier experience will be to use the SOCKS proxy approach as explained here: https://cloud.google.com/dataproc/cluster-web-interfaces

In that case, simply using gcloud compute ssh to run a lightweight local socks proxy and then opening a browser pointed at that will let you click all the YARN links as normal.

Spark UI for finished jobs will also be available under the "History" link on the job entry. — cerisier, Jul 07 '16 at 21:52
On another note, like hortonworks and cloudera both provide an UI interface, does dataproc has any ? — Koushik Paul, Feb 03 '19 at 14:57
Replacing the hostname with private IP doesn't work for some reason.... — pavbagel, May 11 '20 at 19:42

score 3 · Answer 2 · edited Feb 20 '18 at 01:39

3

When following the instructions in Dennis's answer, I found that I could not connect to ports 8080 or 8088 for dataproc image v1.0.

The open ports on the master node suggested to use 18080, which I did following the documentation for port 18080 and voilá: Access to webui.

edited Feb 20 '18 at 01:39

Shog9

156,901
35
231
235

answered Mar 12 '16 at 17:03

Frank

406
2
13

score 0 · Answer 3 · answered Nov 16 '18 at 08:58

0

Since I had public addresses in my DataProc cluster I created a Firewall rule in Cloud Console from my corporate subnet to DataProc instances ports 8088 (YARN RM) and 8042 (NM Webapp address).

answered Nov 16 '18 at 08:58

gogasca

9,283
6
80
125

score 0 · Answer 4 · answered Aug 17 '23 at 16:37

For accessing the Spark UI you can follow these steps:

First, don't forget to mark the Enable component gateway option in the Components section of the Set up cluster tab while creating the Dataproc cluster.
After the cluster is running, go to the Dataproc page in the GCP Console: Dataproc -> Clusters
Find the cluster in the list and click on its link under the Name column for accessing the Cluster Details page.
In the Cluster Details page click on the Web Interfaces tab.
In the Web Interfaces tab click on the YARN ResourceManager link.
The browser will open a new page for the YARN ResourceManager showing all available applications.
Find the application with the type SPARK and the state RUNNING in the list/table on the YARN ResourceManager page.
Scroll right and click on the link ApplicationMaster in the Tracking UI column.
After this, you'll be forwarded to the Spark UI page.

Where is the Spark UI on Google Dataproc?

4 Answers4

Linked