Spark submit jobs fails with no error in local mode

Asked Sep 04 '19 at 14:13

Active Sep 10 '19 at 18:36

Viewed 820 times

I have a spark application that I am trying to execute on windows/Unix using below command

spark-submit --class com.myorg.dataquality.DataVerification  --master local[*] C:\Users\workspace\dataQuality\target\data-quality-framework-0.0.1-SNAPSHOT.jar

The job is getting terminated immediately after submission. However it is working perfect in eclipse.

C:\Users>spark-submit --class com.myorg.dataquality.DataVerification  --master local[*] C:\Users\workspace\dataQuality\target\data-quality-framework-0.0.1-SNAPSHOT.jar
        19/09/04 19:33:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        log4j:WARN No appenders could be found for logger (org.apache.spark.deploy.SparkSubmit$$anon$2).
        log4j:WARN Please initialize the log4j system properly.
        log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
C:\Users>

What could be the issue with it:

Spark Version 2.4.4
Scala Version 2.11.11

I am able to run the job using eclipse, however i am unable to execute it on windows/Linux standalone cluster. If I try spark-shell it is initiating properly.

edited Sep 10 '19 at 18:36

asked Sep 04 '19 at 14:13

Sandeep Singh

7,790
4
43
68

1

Could you please tell what ```DataVerification``` does – Rakshith Sep 04 '19 at 14:24
Make sure all of your app's dependencies are on the classpath, or use an uber jar. – Travis Hegner Sep 04 '19 at 16:52
Have you locate the WinUtils.exe to `%HADOOP_HOME%/bin`? – Lamanus Sep 05 '19 at 02:06
@TravisHegner, Yes, I am using jar with dependencies, however I still see the same issue. – Sandeep Singh Sep 10 '19 at 10:29

Spark submit jobs fails with no error in local mode

0 Answers0

Linked