I know Apache Livy is the rest interface for interacting with spark from anywhere. So what is the benefits of using Apache Livy instead of spark-jobserver. What are the drawbacks of spark-jobserver for which Livy is used as an alternative. And I…
I'm building a RESTful API on top of Apache Spark. Serving the following Python script with spark-submit seems to work fine:
import cherrypy
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('myApp').getOrCreate()
sc =…
I have a dockerized Spark instance and use SJS to submit spark jobs from my Spring boot application.
Everything worked fine so far but now I get the following exception when submitting a job as per the sjs log.
Uncaught exception while reverting…
We are trying to find a way to load a Spark (2.x) ML trained model so that on request (through a REST interface) we can query it and get the predictions, e.g. http://predictor.com:8080/give/me/predictions?a=1,b=2,c=3
There are libs out-of-box to…
I'm using the spark-submit command I have for the log4j properties to invoke a Spark-submit like this:
/opt/spark-1.6.2-bin-hadoop2.6/bin/spark-submit \
--driver-java-options \
"-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties\…
I am a new user of Spark. I have a web service that allows a user to request the server to perform a complex data analysis by reading from a database and pushing the results back to the database. I have moved those analysis's into various Spark…
I'm getting weird error whenever I re-create (delete and create context) the Spark SQL Context and run the job for 2nd time or after it will always throw this exception.
[2016-09-20 13:52:28,743] ERROR .jobserver.JobManagerActor []…
I just will try to explain my simplified use case. There is:
Spark Application which count the words.
A web server which serves a web page with a form.
User that can type word within this form and submit it.
Server receives the word and sends it…
I am using Spark Job Server (SJS) to create context and submit jobs.
My cluster includes 4 servers.
master1: 10.197.0.3
master2: 10.197.0.4
master3: 10.197.0.5
master4: 10.197.0.6
But only master1 has a public ip.
First of all I set up zookeeper…
I have simple spark streaming application which reads data from Kafka and then send this data after transformation on a http end point (or another kafka - for this question let's consider http). I am submitting jobs using job-server.
I am currently…
I create a spark SQL job with spark job server and use HiveContext following the sample below:
https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-extras/src/spark.jobserver/HiveTestJob.scala
I was able to start the server but…
I did the build using sbt job-server-api/packages and sbt job-server-tests/packages.
I am able to extend/unzip the jar using jar -cxf command
curl --data-binary…
We create a system consisting of multiple Spark Streaming applications with each applications having multiple receivers. As far as I understood, each receivers needs its own core in the cluster. We need multiple receivers to accommodate peaks but we…
I'm using Spark Job Server to run a Spark job and it works perfectly. But when I try to execute a big job (needs more than 40 sec) I get this error:
The server was not able to produce a timely response to your request.
Is there some configuration…
