4

I just will try to explain my simplified use case. There is:

  1. Spark Application which count the words.
  2. A web server which serves a web page with a form.
  3. User that can type word within this form and submit it.

Server receives the word and sends it to the Spark Application. Spark application takes as an input this word, based on some data and this word launches a job with recalculations. Once Spark done with calculations, it sends results to web server which shows results on a web page.

The question is, how i can establish communication between spark application and web-server?

I guess, that spark-jobserver or spark-streaming can help me here, but i am not sure about it.

Oleg Yarin
  • 161
  • 2
  • 12
  • thank to @user398371, i make it work. For those, who are interested, can take a look on [wordCountExample](https://github.com/monapasan/apache-spark-job-server-with-web-app) that are builded with [scalatra](https://github.com/scalatra/scalatra) and [Spark Job Server](https://github.com/spark-jobserver/spark-jobserver) – Oleg Yarin Jun 18 '16 at 16:36

1 Answers1

3

There are a few projects that will help you with this.

Generally you run a separate webserver for managing the spark jobs as there is some messy systemExec work around the spark-submit cli to accomplish this. Obviously this runs on a different port than you're primary application and is only accessible by the server component of the primary web application.

There are a few open source projects that will handle this for you most notably:

https://github.com/spark-jobserver/spark-jobserver

https://github.com/cloudera/livy

user398371
  • 857
  • 2
  • 10
  • 15