8

I am trying to install Spark in Win7 and am getting an error.

These are my environment settings:

SPARK_HOME : C:\spark (this is where I have unzipped the spark files)

JAVA_HOME : C:\Program Files\Java\jdk1.7.0_71;

SCALA_HOME: C:\Program Files (x86)\scala

PATH : C:\Program Files\Java\jdk1.7.0_71\bin;C:\app\Sampad\product\11.2.0\dbhome_1\bin;C:\Python27;C:\Program Files\Python27\;C:\Program Files\Python27\Scripts;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Java\jdk1.6.0_45\bin;C:\Program Files\nodejs\;C:\Program Files\Python27;C:\Anaconda;C:\Anaconda\Scripts;C:\HashiCorp\Vagrant\bin;C:\Program Files (x86)\scala\bin;C:\spark\bin;

HADOOP_HOME : C:\winutils; (set this after reading this blog

Please let me know what mistake I have made. Thanks in advance.

Sampada
  • 2,931
  • 7
  • 27
  • 39
user1548787
  • 101
  • 1
  • 1
  • 8
  • Please someone let me know what mistake I have done. I have already searched many sites and tried everything,but it is not working. – user1548787 Nov 16 '15 at 17:46
  • you did not state what error you are getting. What happens when you type 'spark-shell' at a command line? – aquagremlin Dec 23 '15 at 03:08
  • Please check this answer stackoverflow.com/a/52831841/2516356 – Moustafa Mahmoud Oct 16 '18 at 09:08
  • I ran into the same issue, in general go through your PATH and verify that you're not having any of the SPARK, SCALA and JAVA paths set incorrectly - while following tutorials to set up the environment it's easy to add an extra "\" or "\bin" by mistake. – Pio Sep 12 '19 at 17:37

10 Answers10

7

I installed java and spark in folder with out spaces and I am using windows 10.

In my case I added

JAVA_HOME=C:\Java\jdk1.8.0_191\bin

so it was trying to search for executables in "C:\Program Files\Java\jdk1.8.0_191\bin\bin"

make sure to add variable as

JAVA_HOME = C:\Java\jdk1.8.0_191

and In path environmental variable add

%JAVA_HOME%\bin

""same for SPARK_HOME""

It is working for me now !!!

Hamsavardhini
  • 101
  • 2
  • 7
3

Probably, it happens, because different java versions:

JAVA_HOME : C:\Program Files\Java\jdk1.7.0_71;

C:\Program Files\Java\jdk1.6.0_45\bin

Instead of "C:\Program Files\Java\jdk1.6.0_45\bin" use "%JAVA_HOME%\bin"

Spark: Trying to run spark-shell, but get 'cmd' is not recognized as an internal or

Community
  • 1
  • 1
constructor
  • 1,412
  • 1
  • 17
  • 34
  • For me, it was the fact that JAVA_HOME was wrong but it was correct on the PATH, so "java -version" worked properly. Fixing it fixed the issue. – pbahr Mar 28 '18 at 20:17
3

Try modifying the spark-shell2.cmd file first line from 

@echo off

to

rem @echo off

For me it showed me that it was trying to load a file from c:\spark\bin\bin on the following line 

"%SPARK_HOME%\bin\spark-submit2.cmd" --class org.apache.spark.repl.Main --name "Spark shell" %*

In my env %SPARK_HOME% was set to c:\spark\bin. My installation is at c:\spark\bin

So I set the %SPARK_HOME% to c:\Spark and added %SPARK_HOME%\bin to my PATH.

ThatDataGuy
  • 1,969
  • 2
  • 17
  • 43
2

you need to setup JAVA_HOME and path the first thing is you have to go

edit system Environment variable -> Environment variable -> under the User varibales for <user>

add New 

1. JAVA_HOME = C:\Program Files\Java\jdk1.8.0_191
In path 
add %JAVA_HOME%\bin

2. SPARK_HOME = C:\spark   //*spark is my folder where i have installed spark*
In path
add %SPARK_HOME%\bin

This will fix your problem

Sai Mammahi
  • 217
  • 2
  • 14
1

For me everything that is specified above was correct still it was not working.

The reason I found is all the environment variables where added in "system variables" section so when I added them in "user variables" section it started working.

I added JAVA_HOME,SPARK_HOME, HADOOP_HOME and PATH in user variables.

MrMaavin
  • 1,611
  • 2
  • 19
  • 30
AnupKurup
  • 41
  • 4
1

I had the same problem, but I solved it by correctly setting up JAVA_HOME environment variable. Basically, you need JAVA version 8 to run Spark. So you need to install JAVA 8 and properly set up the environment and the path just like the way you set up SPARK_HOME. For example variable name: JAVA_HOME, variable value: C:\JAVA. Then you go to the path and set up the path for java as: %JAVA_HOME%

MindNeedsBook
  • 11
  • 1
0

I had similar issue, I reinstalled java(new version) and corrected JAVA_HOME. This resolved the issue for me

vpv
  • 339
  • 1
  • 2
  • 7
0

I had same issue on windows 10, make sure that only JAVA_HOME value is absolute path, rest everything is relative to %JAVA_HOME%

Sunil Yadav
  • 59
  • 6
0

One of the reasons is either JAVA_HOME or SPARK_HOME has a space in the path. In this case :

"SCALA_HOME: C:\Program Files (x86)\scala"

Here after "Program Files (x86)" has space in its path. Try to move files to location where there won't be space involved in the full path. In my case I had space in JAVA_HOME path.

Yeshwanth N
  • 570
  • 4
  • 15
0

In my case I noticed my JAVA_HOME env variable had semicolon(;) at the end of the path resulting in above mentioned error (as this was leading to incorrect path resolution). I could find this issue by commenting @echo off.

user16861377
  • 1