0

Update 1: After adding missing pieces and env variables from Spark installation - Error: Could not find or load main class org.apache.spark.launcher.Main, the command no longer throws an error, but prints itself and doesn't do anything else. This is the new result of running the command:

"C:\Program Files\Java\jdk1.8.0_271\bin\java" -cp "C:\Users\xxx\repos\spark/conf\;C:\Users\xxx\repos\spark\assembly\target\scala-2.12\jars\*" org.apache.spark.deploy.SparkSubmit --master k8s://http://127.0.0.1:8001 --deploy-mode cluster --conf "spark.kubernetes.container.image=xxx.azurecr.io/spark:spark2.4.5_scala2.12.12" --conf "spark.kubernetes.authenticate.driver.serviceAccountName=spark" --conf "spark.executor.instances=3" --class com.xxx.bigdata.xxx.XMain --name xxx_app https://storage.blob.core.windows.net/jars/xxx.jar

I have been following this guide for setting up Spark in AKS: https://learn.microsoft.com/en-us/azure/aks/spark-job. I am using Spark tag 2.4.5 with scala 2.12.12. I have done all the following steps:

  • created AKS with ACR and Azure storage, serviceaccount and role
  • built spark source
  • built docker image and push to ACR
  • built sample SparkPi jar and push to storage
  • proxied api-server (kubectl proxy) and executed spark-submit:
./bin/spark-submit \
    --master k8s://http://127.0.0.1:8001 \
    --deploy-mode cluster \
    --name xxx_app\
    --class com.xxx.bigdata.xxx.XMain\
    --conf spark.executor.instances=3 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.container.image=xxx.azurecr.io/spark:spark2.4.5_scala2.12.12 \
    "https://storage.blob.core.windows.net/jars/xxx.jar"

All I am getting is Error: Could not find or load main class org.apache.spark.launcher.Main

Now, the funny thing is that it doesn't matter at all what I change in the command. I can mess up ACR address, spark image name, jar location, api-server address, anything, and I still get the same error.

I guess I must be making some silly mistake as it seems nothing can break the command more than it already is, but I can't really nail it down. Does someone have some ideas what might be wrong?

Rafal Kural
  • 57
  • 5

2 Answers2

0

Looks like it might be a problem on the machine you are executing spark-submit. You might be missing some jars on the classpath on the machine you are executing spark-submit. Worth checking out Spark installation - Error: Could not find or load main class org.apache.spark.launcher.Main

Syam Sankar
  • 361
  • 1
  • 6
  • According to the link, the microsoft guide missed on a bunch of prep steps for spark. I have added a separate installation for spark following the official spark guide and now got to the point, where it doesn't throw an error, but still doesn't work and, instead of submitting job, prints out the command to the console without doing anything. Updated main post – Rafal Kural Nov 18 '20 at 09:59
  • Can you share the console output here? – Syam Sankar Nov 19 '20 at 05:43
0

Alright, so I managed to submit jobs with spark-submit.cmd, instead. It works, without any additional setup.

I didn't manage to get the bash script to work in the end and I do not have the time to investigate it further at this moment. So, sorry for providing a half-assed answer only partially resolving original problem, but it is a solution nonetheless.

The below command works fine

bin\spark-submit.cmd --master k8s://http://127.0.0.1:8001 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.namespace=dev --conf spark.kubernetes.container.image=xxx.azurecr.io/spark:spark-2.4.5_scala-2.12_hadoop-2.7.7  https://xxx.blob.core.windows.net/jars/SparkPi-assembly-0.1.0-SNAPSHOT.jar
Rafal Kural
  • 57
  • 5