0

I am facing and issue when trying to read data form Mongo DB.

I have Spark 1.6.3 which has Scala 2.10.5

I am using the Mongo DB Connector Version 1.1 and package 2.10

Following is the dependencies i had used in my Mavan

<dependency> 
<groupId>org.mongodb.spark</groupId> 
<artifactId>mongo-spark-connector_2.10</artifactId> 
<version>1.1.0</version> 
</dependency>

Getting error like 

java.lang.NoClassDefFoundError: com/mongodb/spark/rdd/api/java/JavaMongoRDD
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:708)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.rdd.api.java.JavaMongoRDD
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

But was able to find the class existing in the reference

Thanks in Advance

Sam Berchmans
  • 127
  • 13
  • Is there any classnotfound exception ? – QuickSilver Jun 17 '20 at 17:16
  • @QuickSilver : Yes the below , ` java.lang.NoClassDefFoundError: com/mongodb/spark/rdd/api/java/JavaMongoRDD Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.rdd.api.java.JavaMongoRDD at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ` – Sam Berchmans Jun 17 '20 at 17:33
  • @QuickSilver - I have added a comment , also added in the Post. – Sam Berchmans Jun 17 '20 at 17:38

2 Answers2

1

Add mongo-spark-connector_2.10's jar to your run time class path

It can be done it followting ways,

  • Add mongo-spark-connector jar to Spark lib folder
  • spark-submit with --jars=/path/mongo-spark-connector.jar
  • spark.driver.extraClassPath or it's alias --driver-class-path to set extra classpaths on the node running the driver.
  • spark.executor.extraClassPath to set extra class path on the Worker nodes.

If you add Driver class path then you will have need to add executor class paths to make jar available to executor on worker nodes

Try to use the latest version jar 2.2.8 which shall have the class mongo-spark-connector_2.10:2.2.8

QuickSilver
  • 3,915
  • 2
  • 13
  • 29
  • How should i do it ? Please do assist as i am learning thing in here – Sam Berchmans Jun 17 '20 at 17:42
  • I am running as following, spark-submit --class com.LoadData.LoadFromMongoDB --master yarn path/LoadDataSparkModules-0.0.1-SNAPSHOT.jar This is how i execute it – Sam Berchmans Jun 17 '20 at 17:50
  • can u refer this https://stackoverflow.com/questions/37132559/add-jars-to-a-spark-job-spark-submit – QuickSilver Jun 17 '20 at 18:09
  • i tried running using the option --driver-class-path , but still the same issue. – Sam Berchmans Jun 17 '20 at 18:33
  • @SamBerchmans Addew few options to add jars. Please check – QuickSilver Jun 18 '20 at 03:12
  • :- I tried the following option , but still facing the same issue. `spark-submit --class com.virtualpairprogrammers.LoadFromMongoDB --master yarn /SparkJarPath/DataLoading-0.0.1-SNAPSHOT.jar --jar /jarpath/mongo-spark-connector_2.10-1.1.0.jar --driver-class-path=/jarpath/mongo-spark-connector_2.10-1.1.0.jar` Above option i tried , but still it failed with the same error. Also Tried the same without Jar and other options, but still the same options. – Sam Berchmans Jun 18 '20 at 19:24
  • Use Mongo Connector 2.2.8 ? Because i am not sure if that MongoConnector is good with 1.6.3 . Let me check the same. – Sam Berchmans Jun 18 '20 at 19:57
  • :- Tried to read the documentation for 2.2.8 , that does not support Spark 1.6.3 version. MongoConnector 1.1.0 is the version which supports 1.6.3 – Sam Berchmans Jun 19 '20 at 19:47
0

The reason for this issue is,

When i am trying to build using Maven , i am not using Fat Jar build using tools.

Instead of Maven build i tried the following to build a FAT Jar using Eclipse File -> Export -> Runnable Jar -> Select Class , Jar file Name and choose Option 1 in Library handling. Then click finish.

This is a old type of Jar , but it fixed the issue for me.

Thanks all for the assistance.

Sam Berchmans
  • 127
  • 13