1

I'm trying to install Apache Spark on Windows 10. I downloaded Spark and winutils.exe, set the SPARK_HOME, HADOOP_HOME, and updated the PATH variable to include Spark bin path. Still, when I run spark-shell I get the error below. What's the problem?

C:\tools\spark-2.1.1-bin-hadoop2.7\bin>spark-shell
'""C:\Program' is not recognized as an internal or external command,
operable program or batch file.
ps0604
  • 1,227
  • 23
  • 133
  • 330
  • You have spaces in your PATH – OneCricketeer Jun 02 '17 at 23:50
  • Yes, I have spaces in C:\Program Files\Java\jdk1.8.0_92\bin in PATH, that's the problem? should I append quotes? – ps0604 Jun 02 '17 at 23:54
  • It looks like quote marks might help, either too many or two few. Can you add the install location, and the exact values of SPARK_HOME, HADOOP_HOME and PATH to your question. The spark-shell script adds quote marks to SPARK_HOME so it might be worth trying it WITHOUT quotes (if you've got them). – GregHNZ Jun 03 '17 at 00:45
  • Try https://stackoverflow.com/questions/892555/how-do-i-specify-c-program-files-without-a-space-in-it-for-programs-that-cant – OneCricketeer Jun 03 '17 at 03:27
  • 1
    Possible duplicate of [Why does spark-shell fail with "'""C:\Program' is not recognized as an internal or external command" on Windows?](https://stackoverflow.com/questions/44066854/why-does-spark-shell-fail-with-c-program-is-not-recognized-as-an-internal) – Jacek Laskowski Jun 03 '17 at 10:38

2 Answers2

2

What I have figured out after trying for long time and going through different articles, that this issue is related to setting up the environment variables correctly.Things are actually simple and just need to get the set up right to see your spark shell working, below are the steps mentioned to get it right and working.

  1. Install Java (1.7+) under "C" directory or under a directory where spaces our not there between the full path. Like I have installed java 1.8 version under "C:/Java" path. But if you have already installed java under "Program Files"/"Program Files(86)", you need to put both JAVA_HOME and PATH Variables with double quotes, like JAVA_HOME="C:\Program Files\Java" and PATH="%JAVA_HOME\bin%".

    C:\Users\ankitthakur>java -version java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode).

  2. Install Scala under "C:\Scala" just for consistency or you can choose any other directory. Set SCALA_HOME and add into PATH variable.

    C:\Users\ankitthakur>scala -version Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPF

  3. Install SBT under "C:\Sbt" and similarly set up the SBT_HOME and PATH.

  4. Download Spark from the below link. please remember to download a pre-built version for Hadoop otherwise you need to build your downloaded source code and you can do this through maven if you have installed already otherwise download and install it. Place it under "C:\Spark" directory just for consistency and set up SPARK_HOME and PATH. Donwload Path : http://spark.apache.org/downloads.html

  5. We are done here, but in windows system we get error if we dont install winutils.exe file while running the spark-scripts. so deal with this situation download it from below mentioned path and put it under "C:\Winutils\bin" directory{we need to create this directory} and set HADOOP_HOME and PATH variables.

finalllyyyy!!!!!, we are done here, it seems to lengthy but it is not actually believe me, but make sure you have everything properly in place. Please have a loot at the environment variables snapshot on my machine. environment variables set for setting up spark.

thankit95
  • 36
  • 6
0

i had the same problem. just take that spark folder and paste it in C: and specify the path in the environment variable. It should work.

kailash pandey
  • 11
  • 2