4

I am trying to run a pyspark program that access the hive server. The program terminates by throwing the error

pyspark.sql.utils.AnalysisException: 'java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxrwxr-x;

Now, The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: -wx------ suggests that I should chmod it 777. I did.

drwxrwxrwx - hadoopuser supergroup 0 2019-07-03 06:39 /tmp/hive

However, pyspark still throws the same error.

Because of that, I decided to change the scratch dir folder to /tmp/sparkpy. I registered this change at hive-site.xml

I restarted the hiveserver2 and Beeline confirmed that the change is registered.

+------------------------------------+
|                set                 |
+------------------------------------+
| hive.exec.scratchdir=/tmp/sparkpy  |
+------------------------------------+
1 row selected (0.028 seconds)

However, pyspark is still stuck at /tmp/hive and insists that it's not writeable.

DennisLi
  • 3,915
  • 6
  • 30
  • 66
Realdeo
  • 449
  • 6
  • 19

1 Answers1

0

I was also facing this same issue. Finally, I got the answer /tmp/hive location is present in your local system.

sudo chmod -R 777 /tmp

Above command fixed issue for me.

Krishna Kumar Singh
  • 166
  • 1
  • 16