1

I have the following code to connect to hive from pyspark. But i'm getting this error,

AnalysisException: 'java.lang.RuntimeException: java.io.IOException: (null) entry in command string: null chmod 0700 C:\Users\deepak\AppData\Local\Temp\006a8f0d-4fde-473a-951d-ae1b6c3f1e45_resources;' This is what i'm trying to connect.

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession, HiveContext,SQLContext
SparkContext.setSystemProperty("hive.metastore.uris", "thrift://x.x.x.x:9083")
SparkContext.setSystemProperty("spark.hadoop.fs.defaultFS", "hdfs://x.x.x.x:8020")
SparkContext.setSystemProperty("spark.yarn.jars", "hdfs://x.x.x.x/user/talentorigin/jars/*.jar")
SparkContext.setSystemProperty("spark.hadoop.yarn.resourcemanager.address", "x.x.x.x:8032")

sparkSession = (SparkSession
                .builder
                .appName('Pyspark Read from hive')
                .enableHiveSupport()
                .getOrCreate())
df_load=sparkSession.sql('Show databases')
df_load.show()
DEEPAK K M
  • 63
  • 2
  • 7
  • Please check this question https://stackoverflow.com/questions/40764807/null-entry-in-command-string-exception-in-saveastextfile-on-pyspark It seems a problem related to winutils.exe – Diego Rodríguez Apr 09 '19 at 13:31

0 Answers0