1
df_export=(
    spark.table('db.table')
)

df_new_df=df_export.orderBy("count")

both df_export and df_new_df is stored in spark memory and not in files. Definition of Tempory view in spark is "A Temporary view in Spark is similar to a real SQL table that contains rows and columns but the view is not materialized into files.". How is this different from storing in a dataframe as above?

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
Blue Clouds
  • 7,295
  • 4
  • 71
  • 112

1 Answers1

0

You need a DataFrame in order to generate a temporary view. As you mentioned when spark.table('table_name') gets invoked data from the data source it is loaded into memory. When you create a temporary view of the data frame, it is just an identifier to be used for the DAG of the DataFrame, nothing is actually stored in memory or on disk.

Reference: https://stackoverflow.com/a/55698546

Brian Huynh
  • 26
  • 3