2

I'm trying to initialize my Datastax graph using Spark as follow :

val graphBuilder = spark.dseGraph("GRAPH_NAME")

but I have the following exception :

Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)

I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.

Any help is really appreciated. Thanks in Advance!

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
BeginMoh
  • 123
  • 1
  • 11
  • what version of DSE, how do you submit application? – Alex Ott Nov 20 '18 at 16:29
  • version Dse : 6.0.4 , i tested with 5.0.4 also – BeginMoh Nov 20 '18 at 16:47
  • how do you submit job? via `dse spark-submit` or you're using separate spark? – Alex Ott Nov 20 '18 at 16:49
  • i'm using a separet spark – BeginMoh Nov 21 '18 at 08:36
  • Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames? – Alex Ott Nov 21 '18 at 09:17
  • Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ? – BeginMoh Dec 07 '18 at 13:30
  • Just checked the BYOS, it has that class - you either don't submit job correctly, or don't link correctly – Alex Ott Dec 07 '18 at 13:41
  • Yeh it exist in jar file , after executing jobs i have this :org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:193) at com.datastax.bdp.graph.spark.DseGraphRpc.callGetSchema(DseGraphRpc.java:50) – BeginMoh Dec 07 '18 at 13:56

1 Answers1

1

To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):

<dependency>
  <groupId>com.datastax.dse</groupId>
  <artifactId>dse-byos_2.11</artifactId>
  <version>6.0.4</version>
</dependency>

And add DataStax repositories:

<repositories>
    <repository>
      <id>DataStax-Repo</id>
      <url>https://repo.datastax.com/public-repos/</url>
    </repository>
</repositories>

And if you check the jar fetched from DataStax repository, it contains necessary file:

unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
     2839  10-06-2018 15:22   com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot – BeginMoh Nov 21 '18 at 13:59
  • Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ? – BeginMoh Dec 07 '18 at 13:39