Pyspark data frame to Hive Table

Question

How to store a Pyspark DataFrame object to a hive table , "primary12345" is a hive table ? am using the below code masterDataDf is a data frame object

masterDataDf.write.saveAsTable("default.primary12345")

getting below error

: java.lang.RuntimeException: Tables created with SQLContext must be TEMPORARY. Use a HiveContext instead.

Possible duplicate of [How to save DataFrame directly to Hive?](https://stackoverflow.com/questions/30664008/how-to-save-dataframe-directly-to-hive) — desertnaut, Nov 09 '17 at 12:17

score 1 · Answer 1 · answered Nov 09 '17 at 12:19

You can create one temporary table.

masterDataDf.createOrReplaceTempView("mytempTable")

Then you can use simple hive statement to create table and dump the data from your temp table.

sqlContext.sql("create table primary12345 as select * from mytempTable");

OR

if you want to used HiveContext you need to have/create a HiveContext

import org.apache.spark.sql.hive.HiveContext;

HiveContext sqlContext = new org.apache.spark.sql.hive.HiveContext(sc.sc());

Then directly save dataframe or select the columns to store as hive table

masterDataDf.write().mode("overwrite").saveAsTable("default.primary12345 ");

Thanks this code works but I need to add two more configurations to the spark application — Akhil Sudhakaran, Nov 24 '17 at 12:43

Pyspark data frame to Hive Table

1 Answers1