2

I've successfully run the example Spark workflow ("Copy a file by launching a Spark Java program") provided in the Hue Oozie workflow editor (in the Cloudera 5.5.1 QuickStart VM).

I'm now trying to run it manually using the oozie commandline tool:

oozie job -oozie http://localhost:11000/oozie -config job.properties -run

The workflow XML is basically unchanged - I have copied it to HDFS and have the following job.properties:

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032
oozie.wf.application.path=/user/cloudera/workflows/spark-scala/spark-scala.xml
input=/user/hue/oozie/workspaces/data/sonnets.txt
output=here

The job is accepted and appears in the Hue web dashboard, but is killed after a few seconds, and the logs report:

Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

What is the problem here?

DNA
  • 42,007
  • 12
  • 107
  • 146

1 Answers1

2

Oozie doesn't include the libraries for the Spark action, by default - you need to add the following to the job.properties:

oozie.use.system.libpath=true

(Clicking on the previously successful Hue workflow in the Hue Dashboard, you can select the Configuration tab to see the properties that Hue has provided)

DNA
  • 42,007
  • 12
  • 107
  • 146