Can't start spark-shell on windows 10 Spark 3.2.0 install

Question

Issue

When I try to run spark-shell I get a huge message error that you can see here : https://pastebin.com/8D6RGxUJ

Install

I used this tutorial, but I already have python and java installed. I used spark 3.2.0 instead.

Config : Windows 10

• HADOOP_HOME : C:\hadoop downloaded from https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.0/bin

• JAVA_HOME : C:\PROGRA~2\Java\jre1.8.0_311

• SPARK_HOME : C:\Spark\spark-3.2.0-bin-hadoop3.2

• in path :

  • %SPARK_HOME%\bin
  • %HADOOP_HOME%\bin

Answer 1

My guess is that you have to put winutils.exe in the same folder as the $SPARK_HOME%\bin folder. I discovered that after starting from scratch and following this tutorial!

Answer 2

By following this answer for a similar question, I downgraded from spark 3.2.1 to 3.0.3 and this seems to have solved this problem.

Answer 3

I managed to solve the problem with the following configuration:

• Spark: spark-3.2.1-bin-hadoop2.7
• Hadoop: winutils.exe and hadoop.dll (version 2.7.7 for both)
• JDK: jdk-18.0.1

And I recommend that you put the environment variables in User, not System