0

I try to execute local spark job through airflow task:

spark = (SparkSession
            .builder
            .master("spark://172.22.102.229:7077")
            .appName("Test")
            .getOrCreate())

But get a error:

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

When i replace .master("spark://172.22.102.229:7077") to .master("local") it is working

My spark deployed and i can get web ui at http://172.22.102.229:4040/ address

Марсель Абдуллин
  • 161
  • 1
  • 1
  • 9

1 Answers1

1

If you can access Spark WebUI go to the WebUI and you will see something like this: enter image description here

Use the URL to create your SparkSession. For example:

spark = SparkSession
            .builder
            .master("spark://10.0.6.187:7077")
            .appName("Test")
            .getOrCreate()

Hope this can help you! Best Regards

Paulo Barbosa
  • 139
  • 6
  • yeah thank you, in my case i also need to add SPARK_MASTER_HOST and SPARK_LOCAL_HOST variable and start spark master (conf/start-master.sh) For details here: https://stackoverflow.com/questions/46430106/worker-failed-to-connect-to-master-in-spark-apache – Марсель Абдуллин Dec 20 '22 at 15:32