5

Use case is I wanted to return a dataframe as an object, to rest service. Rest service don't have spark context control. So is there any way where I can perform ANSI queries, like how I perform on registerAsTemptable.
I will pass table names and queries form rest service. Then I should get return something as an object which I can show as a Table on view.

If there is any alternative way, then please suggest that as well. But I wanted to use spark as a base framework.

Kalpesh
  • 694
  • 2
  • 8
  • 28
  • 1
    Possible duplicate of [Best Practice to launch Spark Applications via Web Application?](http://stackoverflow.com/questions/40300222/best-practice-to-launch-spark-applications-via-web-application) –  Dec 11 '16 at 03:21

1 Answers1

3

No, you cannot return DataFrame to Rest service. It won't work outside Spark context.

Spark has no out-of-the box service support

However, you can:

  1. Start JDBC Spark Server and query to this server. Here is tutorial for connecting to this server. It is not REST Service, it's just JDBC server. You can connect to it from your REST Services (but not in REST way, just use as data source), use it as data source
  2. submit jobs to Livy Server - your service may call Livy to run some jobs in Spark
  3. submit jobs to Spark REST API - your service may call Livy to run some jobs in Spark, but in this case job files must be in JAR file in the cluster

Both 2nd and 3rd option requires prepared job code. It is not REST service that you can call it with query /get/table/row=1, you must prepare your own service that will submit job with proper calculation.

Conclusions: No, Spark doesn't have built-in REST service to query the data. However it has some options to run pre-defined jobs in REST style and to query data. However, this requires your own services to be built. They must query proper Spark API with predefined job.

If you want just to run SQL queries, consider using JDBC Spark Server as data source of your service.

T. Gawęda
  • 15,706
  • 4
  • 46
  • 61