what's the relationship between Spark rdd and Spark sql?

Question

I am a Spark beginner!And,I'm confused about the relationship between Spark rdd and Spark sql . Whether Spark sql is supposed to converted to Spark rdd in the background?

please refer to the programming guide: http://spark.apache.org/docs/latest/programming-guide.html — mtoto, Oct 11 '16 at 12:59
Possible duplicate of [Difference between DataFrame and RDD in Spark](http://stackoverflow.com/questions/31508083/difference-between-dataframe-and-rdd-in-spark) — , Oct 11 '16 at 13:17

score 1 · Accepted Answer · answered Oct 11 '16 at 21:59

As far as I know, they are sitting atop different engines.

Spark SQL leverages an internal thing called Catalyst which is responsible for generating logical plans for the work and doing performance optimization in relation to codegen.

First, because DataFrame and Dataset APIs are built on top of the Spark SQL engine, it uses Catalyst to generate an optimized logical and physical query plan.

https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html

The RDD api on the other hand, is low level, and apparently does not leverage catalyst.

Very grateful for your answer.This solved my confusion. – liyong Oct 13 '16 at 01:20 — liyong, Oct 13 '16 at 01:20

what's the relationship between Spark rdd and Spark sql?

1 Answers1