0

If the following (working) spark-submit command (for a dotnet for Spark app) was executing a python script, would it still use the same --conf settings? Given a python script name of myapp.py that has no defined function (other than main), what would the --class reference be for a python script?

/opt/spark/bin/spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner \
   --conf "spark.eventLog.enabled=true" \
   --conf "spark.eventLog.dir=file:/usr/bin/spark/hadoop/logs" \
   --master spark://spark:7077 \
   /opt/spark/jars/microsoft-spark-3-1_2.12-2.0.0.jar \
   dotnet myapp.dll "somefilename.txt"
Mike Williams
  • 1
  • 3

1 Answers1

0

For Python applications, simply pass a .py file, no need to mention the class name

/opt/spark/bin/spark-submit \    
--conf "spark.eventLog.enabled=true" \
--conf "spark.eventLog.dir=file:/usr/bin/spark/hadoop/logs" \   
--master spark://spark:7077 \
/your python file path/myapp.py

For further info, you can refer https://spark.apache.org/docs/latest/submitting-applications.html

Karthik
  • 1,143
  • 7
  • 12
  • Would it be the same, functionally, if, for whatever reason, trying to execute the .py script only worked when calling it via, "python myapp.py"? ```/opt/spark/bin/spark-submit \ --conf "spark.eventLog.enabled=true" \ --conf "spark.eventLog.dir=file:/usr/bin/spark/hadoop/logs" \ --master spark://spark:7077 \ python /pathtofile/myapp.py``` – Mike Williams Jan 11 '22 at 22:55
  • dont use python, spark-submit will automatically run myapp.py file – Karthik Jan 12 '22 at 00:41
  • 1
    Thank you @Sri_Karthik – Mike Williams Jan 18 '22 at 14:52