0

Requirement: Trigger a spark job from UI by user action (say submit button click). Once the spark job is finished, the summary of the status has to be displayed in UI.

Design approach: 1. Once the user initiates a job run by clicking the submit button from UI, we will make an insert into a Impala queue table using Impala JDBC. The simplified structure of the queue table is as follows:

JOB_RUN_QUEUE (REQUEST_ID, STATUS, INPUT_PARAM_1, INPUT_PARAM_2, SUMMARY)

The initial request will have STATUS='SUBMIT'

  1. Oozie will be configured to orchestrate the request handling and job spark execution. Once Oozie finds entry into the queue table JOB_RUN_QUEUE with status='SUBMIT' it will pull the arguments from the queue table and trigger the spark job. It will update the status in the queue table to 'IN PROGRESS'. Upon successfull completion it will update the summary and status in the queue table. On failure it will update the status to FAILURE.

  2. UI will read the data from the queue table and display on UI.

Questions: 1. Is there any alternative ad better design approach. 2. D I need to have a queue mechanism to the initial request or can I leverage some inbuilt functionality?

Joyan
  • 41
  • 1
  • 7
  • Possible duplicate of [Best Practice to launch Spark Applications via Web Application?](http://stackoverflow.com/questions/40300222/best-practice-to-launch-spark-applications-via-web-application) – T. Gawęda May 15 '17 at 10:24
  • You could have a button click send a request in the form of a JSON message to a web service API you can build. Then, that web service can use the Oozie API to submit a workflow that launches your spark jar. I've used this design and it's worked quite well. – David Schuler May 15 '17 at 17:37

0 Answers0