0

I'm trying to test the Spark-HBase connector in the GCP context and tried to follow [1], which asks to locally package the connector [2] using Maven (I tried Maven 3.6.3) for Spark 2.4, and leads to following issue.

Error "branch-2.4":

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project shc-core: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: NullPointerException -> [Help 1]

References

[1] https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/tree/master/scala/bigtable-shc

[2] https://github.com/hortonworks-spark/shc/tree/branch-2.4

py-r
  • 419
  • 5
  • 15
  • Try building with Java 8 for "branch-2.4"; https://github.com/davidB/scala-maven-plugin/issues/185 – Ismail Dec 23 '20 at 19:53
  • An option to resolve "branch-2.3" is add the missing dependency to the pom file https://stackoverflow.com/a/15618633/11866133 – Ismail Dec 23 '20 at 20:03
  • To get the full debugging picture, re-try executing your commands with the --verbosity flag https://cloud.google.com/sdk/gcloud#suppressing_prompting_writing_to_the_terminal_and_logging – Ismail Dec 23 '20 at 20:14
  • @Ismail: Thanks ! I could build the jar using Java 8. May I ask you how to correctly add it to the `Dependencies.scala` in the GCP template - I tried a few options without success so far. Thanks ! – py-r Dec 23 '20 at 22:10
  • Can you share the options you tried? – Puteri Dec 24 '20 at 00:39
  • Ok packaging done, but I get another issue when submitting the job: https://stackoverflow.com/questions/65466253/spark-hbase-gcp-template-parsing-catalogue-error. – py-r Dec 27 '20 at 13:29

1 Answers1

2

As suggested in the comments (thanks @Ismail !), using Java 8 works to build the connector:

sdk use java 8.0.275-zulu

mvn clean package -DskipTests

One can then import the jar in Dependencies.scala of the GCP template as follows.

...
val shcCore = "com.hortonworks" % "shc-core" % "1.1.3-2.4-s_2.11" from "file:///<path_to_jar>/shc-core-1.1.3-2.4-s_2.11.jar"
...
// shcCore % (shcVersionPrefix + scalaBinaryVersion) excludeAll(
shcCore excludeAll(
...
py-r
  • 419
  • 5
  • 15